Text-to-speech synthesis method, device, equipment, and storage medium

Abstract

Embodiments of the present invention provide a text-to-speech synthesis method, apparatus, device, and storage medium, relating to the field of artificial intelligence technology. The method comprises: obtaining a phoneme sequence and a speech object identifier corresponding to an input text, masking a phoneme alignment sequence based on a preset masking rate and a preset masking step length to obtain a candidate phoneme sequence, performing unsupervised mapping on the phoneme alignment sequence based on the candidate phoneme sequence, converting the phoneme alignment sequence into an unsupervised phoneme sequence, performing speech synthesis on the unsupervised phoneme sequence, and obtaining the speech content corresponding to the phoneme sequence. This embodiment performs speech synthesis on the input text through unsupervised learning, thereby improving the naturalness and interpretability of the synthesized speech and expanding the application scenarios of speech synthesis technology.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a text-to-speech synthesis method, apparatus, device, and storage medium. Background Art

[0002] Speech synthesis technology plays an important role in scenarios such as human-computer interaction. With the development of deep learning, speech synthesis technology has entered the end-to-end development stage. The end-to-end speech synthesis model can directly output the speech corresponding to the input text based on the text.

[0003] Among related technologies, text-to-speech (TTS) synthesis technology is generally implemented using supervised learning models. Supervised models require labeled data for training. When insufficient prior knowledge is available, the synthesized speech data is not natural or understandable, which greatly limits its application scenarios. For example, in the AI ​​assistant, gaming, or entertainment industries, there is a high demand for customized speech synthesis systems, which supervised models cannot meet. Therefore, how to improve the naturalness and interpretability of synthesized speech has become a pressing technical problem. Summary of the Invention

[0004] The main purpose of the embodiments of the present invention is to propose a text-to-speech synthesis method, apparatus, device and storage medium to improve the naturalness and interpretability of synthesized speech and expand the application scenarios of speech synthesis technology.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present invention provides a text-to-speech synthesis method, comprising:

[0006] Get the phoneme sequence and speech object identifier corresponding to the input text;

[0007] Performing duration prediction on the phoneme sequence according to the speech object identifier to obtain a phoneme alignment sequence;

[0008] Masking the phoneme alignment sequence based on a preset masking rate and a preset masking step size to obtain a candidate phoneme sequence;

[0009] Performing unsupervised mapping on the candidate phoneme sequence to obtain an unsupervised phoneme sequence;

[0010] Speech synthesis is performed on the unsupervised phoneme sequence to obtain speech content corresponding to the phoneme sequence.

[0011] In some embodiments, the step of predicting the duration of the phoneme sequence according to the speech object identifier to obtain a phoneme aligned sequence includes:

[0012] Obtaining a speech object embedding corresponding to the speech object identifier and a phoneme embedding corresponding to the phoneme sequence;

[0013] Extracting attention information from the phoneme embedding using a multi-head attention unit to obtain a phoneme representation vector;

[0014] Performing duration prediction based on the phoneme representation vector and the speech object embedding to generate the duration of each phoneme in the phoneme sequence;

[0015] The phoneme alignment sequence is obtained according to the phoneme sequence and the duration of each phoneme.

[0016] In some embodiments, masking the phoneme alignment sequence based on a preset masking rate and a preset masking step size to obtain a candidate phoneme sequence includes:

[0017] Obtaining a phoneme alignment embedding of the phoneme alignment sequence using preset alignment lookup table information;

[0018] Select the starting position index information;

[0019] The candidate phoneme sequence is obtained by masking the phoneme alignment embedding according to the starting position index information, the preset masking rate and the preset masking step size.

[0020] In some embodiments, performing unsupervised mapping on the candidate phoneme sequence to obtain an unsupervised phoneme sequence includes:

[0021] The candidate phoneme sequence and the phoneme are embedded into a trained bidirectional long short-term memory module for prediction to obtain the phoneme unsupervised sequence.

[0022] In some embodiments, performing speech synthesis on the unsupervised phoneme sequence to obtain speech content corresponding to the phoneme sequence includes:

[0023] Performing vector encoding on the unsupervised phoneme sequence to obtain a hidden feature vector;

[0024] Probabilistically encoding the hidden layer feature vector to obtain speech object distribution and speech content distribution;

[0025] Sampling and encoding the speech object distribution using the speech object embedding to obtain a speech object sampling encoding vector;

[0026] Performing sampling encoding on the speech content distribution using the phoneme embedding to obtain a speech content sampling encoding vector;

[0027] Decoding the speech object sampling coding vector and the speech content sampling coding vector to obtain a spectrogram of the phoneme sequence corresponding to the speech object label;

[0028] The speech content corresponding to the speech object identifier is generated according to the spectrogram.

[0029] In some embodiments, the probability coding includes posterior probability coding or prior probability coding, the speech object distribution includes: speech object posterior distribution or speech object prior distribution, and the speech content distribution includes: speech content posterior distribution or speech content prior distribution.

[0030] In some embodiments, obtaining a phoneme sequence corresponding to the input text includes:

[0031] Converting the input text into word component information;

[0032] The word segmentation information is mapped into the corresponding phoneme sequence using a preset phoneme dictionary.

[0033] To achieve the above objectives, a second aspect of an embodiment of the present application provides a text-to-speech synthesis device, comprising:

[0034] An acquisition unit, configured to acquire a phoneme sequence and a speech object identifier corresponding to an input text;

[0035] a duration prediction unit, configured to predict the duration of the phoneme sequence according to the speech object identifier to obtain a phoneme alignment sequence;

[0036] a sequence masking unit, configured to mask the phoneme alignment sequence based on a preset masking rate and a preset masking step length to obtain a candidate phoneme sequence;

[0037] an unsupervised mapping unit, configured to perform unsupervised mapping on the candidate phoneme sequence to obtain an unsupervised phoneme sequence;

[0038] The speech synthesis unit is used to perform speech synthesis on the unsupervised phoneme sequence to obtain speech content corresponding to the phoneme sequence.

[0039] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0040] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the method described in the first aspect is implemented.

[0041] The text-to-speech synthesis method, apparatus, device, and storage medium proposed in an embodiment of the present invention obtains a phoneme sequence and speech object identifier corresponding to an input text, masks the phoneme alignment sequence based on a preset masking ratio and a preset masking step size to obtain a candidate phoneme sequence, performs unsupervised mapping on the phoneme alignment sequence based on the candidate phoneme sequence, converts the phoneme alignment sequence into an unsupervised phoneme sequence, and performs speech synthesis on the unsupervised phoneme sequence to obtain the speech content corresponding to the phoneme sequence. This embodiment performs speech synthesis on the input text through unsupervised learning, improving the naturalness and interpretability of the synthesized speech and expanding the application scenarios of speech synthesis technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a flowchart of a text-to-speech synthesis method provided by an embodiment of the present invention.

[0043] Figure 2 yes Figure 1 Flowchart of step S110 in .

[0044] Figure 3 yes Figure 1 Flowchart of step S120 in .

[0045] Figure 4 4 is a schematic diagram of a phoneme alignment sequence of a text-to-speech synthesis method provided by an embodiment of the present invention.

[0046] Figure 5 yes Figure 1 Flowchart of step S130 in .

[0047] Figure 6 yes Figure 5 Flowchart of step S132 in .

[0048] Figure 7 4 is a schematic diagram of a candidate phoneme sequence of the text-to-speech synthesis method provided by an embodiment of the present invention.

[0049] Figure 8 Schematic diagram of the coding and decoding model of the text-to-speech synthesis method provided by an embodiment of the present invention.

[0050] Figure 9 yes Figure 1 Flowchart of step S140 in .

[0051] Figure 10 Schematic diagram of a text-to-speech synthesis system provided by an embodiment of the present invention.

[0052] Figure 11 This is a structural block diagram of a text-to-speech synthesis device provided by another embodiment of the present invention.

[0053] Figure 12It is a schematic diagram of the hardware structure of the electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0055] It should be noted that although the functional modules are divided in the device schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order than the module division in the device or the order in the flowchart.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0057] First, some terms involved in this invention are analyzed:

[0058] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0059] Supervised learning: This is a type of machine learning in which a machine is trained using labeled training data and, based on this data, predicts an output. Labeled data means that some input data has been labeled with the correct output. In supervised learning, the model learns each type of data using a labeled dataset. The training data provided to the machine acts as a supervisor, teaching it to correctly predict the output. The goal is to find a mapping function that maps input variables to output variables. After the training process is complete, the model is tested on test data and then predicts the output. Supervised learning can be used for risk assessment, image classification, fraud detection, spam filtering, and other applications.

[0060] Unsupervised learning, a key branch of machine learning, also known as unsupervised learning, extracts information from a distribution without the need for human annotation. It involves performing classification without any labels, eliminating the need for sample data and enabling direct modeling and clustering. When presented with unlabeled data, the machine infers the data's internal structure to categorize features. Unsupervised algorithms process only features and do not manipulate supervisory signals. Unsupervised learning plays a crucial role in machine learning, data mining, biomedical big data analysis, and data science.

[0061] Attention units: Research on attention units first emerged in psychology. When a person sees an image, despite being presented with the larger picture, their attention is focused on a single point. In 2014, Bahdanau et al. first introduced attention units into machine translation. Combining attention units with neural networks improved machine translation accuracy. Subsequently, numerous studies have attempted to combine attention units with neural networks for various tasks, achieving significant advantages over traditional methods to varying degrees.

[0062] Mel spectrogram: A mel spectrum is obtained by applying a Fourier transform to an acoustic signal and then applying a mel-scale transformation. Spectrograms are often very large images. To obtain appropriately sized sound features, the spectrogram is transformed into a mel spectrum by passing it through a mel-scale filter bank. In the mel-frequency domain, there is a linear relationship between the mel-frequency of speech and the human perception of pitch. The mel spectrum is obtained by combining the mel-frequency cepstrum with the spectrogram.

[0063] Embedding: It is a commonly used feature method in the field of deep learning. It maps high-dimensional raw data (such as images, sentences, speech, etc.) to a low-dimensional manifold, making the high-dimensional raw data separable after being mapped to the low-dimensional manifold. This mapping process is called embedding.

[0064] Long Short-Term Memory (LSTM) artificial neural network: A type of RNN (Recurrent Neural Network). LSTM is well-suited for modeling time series data, such as text data. An LSTM model consists of the input word Xt at time t, the cell state Ct, the temporary cell state Ct', the hidden state ht, the forget gate ft, the memory gate it, and the output gate Ot. The LSTM computational process can be summarized as follows: by forgetting information in the cell state and remembering new information, information useful for subsequent calculations is passed on, while useless information is discarded. At each time step, the LSTM outputs a hidden state ht, calculated from the previous hidden state ht-1 and the current input Xt. The hidden state ht is controlled by the forget gate ft, the memory gate it, and the output gate Ot.

[0065] Speech synthesis technology plays an important role in scenarios such as human-computer interaction. With the development of deep learning, speech synthesis technology has entered the end-to-end development stage. The end-to-end speech synthesis model can directly output the speech corresponding to the input text based on the text.

[0066] Among related technologies, text-to-speech (TTS) synthesis technology is generally implemented using supervised learning models. Supervised models require labeled data for training. When insufficient prior knowledge is available, the synthesized speech data is not natural or understandable, which greatly limits its application scenarios. For example, in the AI ​​assistant, gaming, or entertainment industries, there is a high demand for customized speech synthesis systems, which supervised models cannot meet. Therefore, how to improve the naturalness and interpretability of synthesized speech has become a pressing technical problem.

[0067] Based on this, an embodiment of the present invention provides a text-to-speech synthesis method, apparatus, device, and storage medium. The method obtains a phoneme sequence and speech object identifier corresponding to an input text, masks the phoneme alignment sequence based on a preset masking rate and a preset masking step size to obtain a candidate phoneme sequence, performs unsupervised mapping on the phoneme alignment sequence based on the candidate phoneme sequence, converts the phoneme alignment sequence into an unsupervised phoneme sequence, performs speech synthesis on the unsupervised phoneme sequence, and obtains the speech content corresponding to the phoneme sequence. This embodiment performs speech synthesis on the input text through unsupervised learning, improves the naturalness and interpretability of the synthesized speech, and expands the application scenarios of speech synthesis technology.

[0068] The embodiments of the present invention provide a text-to-speech synthesis method, apparatus, device, and storage medium, which are specifically described through the following embodiments. First, the text-to-speech synthesis method in the embodiments of the present invention is described.

[0069] Embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) refers to theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI is the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0070] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0071] The text-to-speech synthesis method provided by an embodiment of the present invention relates to the field of artificial intelligence technology, and in particular to the field of data mining technology. The text-to-speech synthesis method provided by an embodiment of the present invention can be applied to a terminal, can be applied to a server side, or can be a computer program running in a terminal or a server side. For example, a computer program can be a native program or software module in an operating system; it can be a local (Native) application (APP, Application), that is, a program that needs to be installed in an operating system to run, such as a client that supports text-to-speech synthesis; it can also be a small program, that is, a program that only needs to be downloaded to a browser environment to run; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form. Among them, the terminal communicates with the server via a network. The text-to-speech synthesis method can be executed by a terminal or a server, or by a terminal and a server in collaboration.

[0072] In some embodiments, the terminal may be a smartphone, tablet computer, laptop computer, desktop computer, or smartwatch, etc. In addition, the terminal may also be an intelligent vehicle-mounted device. The intelligent vehicle-mounted device applies the text-to-speech synthesis method of this embodiment to provide related services to enhance the driving experience. The server may be an independent server, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; or a service node in a blockchain system, wherein each service node in the blockchain system forms a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). The server may be installed with a server end of the text-to-speech synthesis system, through which the terminal may interact, for example, by installing corresponding software on the server end. The software may be an application that implements the text-to-speech synthesis method, etc., but is not limited to the above forms. The terminal and the server may be connected via Bluetooth, USB (Universal Serial Bus), a network or other communication connection methods, which are not limited in this embodiment.

[0073] The present invention can be used in a wide variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0074] To facilitate understanding of the embodiments of the present application, the concept of text-to-speech synthesis is first briefly introduced below with reference to examples of specific application scenarios.

[0075] Text-to-speech synthesis is an important part of human-computer voice communication. Using text-to-speech synthesis technology, machines can speak like humans, so that some information represented or stored in text form can be converted into speech, so that people can easily obtain this information through hearing.

[0076] In one application scenario, user A sends a text message and can select a voice object identifier (mainly used to identify different timbres, such as a celebrity or other robotic voices) based on their needs. User B on the other end receives the voice message corresponding to the text message and, when playing the voice message, actually hears the selected timbre. This process uses text-to-speech synthesis to convert user A's text message into a voice message with the desired timbre.

[0077] Input: text message sent by user A and the selected voice object identifier;

[0078] Output: Converted into the target speech corresponding to the speech object identifier.

[0079] Reference Figure 1 , is an optional flowchart of the text-to-speech synthesis method provided by an embodiment of the present invention, Figure 1 The method may include but is not limited to steps S110 to S150.

[0080] Step S110: Acquire the phoneme sequence and voice object identifier corresponding to the input text.

[0081] In some embodiments, reference Figure 2 , step S110 obtains the phoneme sequence corresponding to the input text, including but not limited to steps S111 to S112.

[0082] Step S111: convert the input text into word component information.

[0083] Step S112: Map the word segmentation information into a corresponding phoneme sequence using a preset phoneme dictionary.

[0084] In one embodiment, if the input text is Chinese characters in Chinese, Chinese characters are usually composed of initial consonants and final vowels, which can be further subdivided according to the principles of linguistics. The same final vowel and different initial consonant combinations can be expressed as different phonemes. Therefore, the initial consonants and final vowels are defined as phoneme sets through linguistic principles and a constructed preset phoneme dictionary.

[0085] In one embodiment, initials and finals are defined as 66 types of phonemes to form a preset phoneme dictionary, which is represented as follows:

[0086] "a", "aa", "ai", "an", "ang", "ao", "b", "c", "ch", "d", "e", "ee", "ei", "en", "eng", "er", "f" ,"g","h","i","ia","ian","iang","iao","ie","ii","in","ing","iong","iu","ix","iy"," iz", "j", "k", "l", "m", "n", "o", "ong", "oo", "ou", "p", "q", "r", "s", "sh", "t", "u", "ua", "ua i", "uan", "uang", "ueng", "ui", "un", "uo", "uu", "v", "van", "ve", "vn", "vv", "x", "z", "zh".

[0087] In one embodiment, if the input text is in a language other than Chinese, elements in the input text are subdivided into different phonemes according to phonetic principles. For example, in one embodiment, the input text is in English. The smallest phonetic units in English are vowels and consonants. Therefore, vowels and consonants are defined as phoneme sets in a preset phoneme dictionary constructed according to linguistic principles.

[0088] In one embodiment, vowels and consonants are defined as 48 types of phonemes to form a preset phoneme dictionary, which is represented as follows:

[0089] "p","m","w","h","t","d","n","j","r","E:","E","k","g","N","l","C:","C","f","v",

[0090] "s", "z", "Q", "W", "T", "e", "A", "F", "V", "ai", "ei", "Ci", "tF", "dV", "ZE", "tr", "dr", "Eu", "au", "ts", "dz".

[0091] As can be seen from the above, the text content of the input text is transcribed into word segmentation information, and then the corresponding preset phoneme dictionary is selected according to the language. The pinyin is split into individual phonemes according to the preset phoneme dictionary, and the individual phonemes are spliced ​​into a phoneme sequence to facilitate subsequent processing of the phoneme sequence.

[0092] The text-to-speech synthesis method of the embodiment of the present disclosure does not limit the language of the input text. For the convenience of description, the following description is given by taking the input text as Chinese and the word segmentation information as Pinyin as an example, which does not mean that the present disclosure limits the language of the input text.

[0093] In one embodiment, after the sound sequence is obtained by the above steps, the voice object identifier selected by the user is obtained at the same time.

[0094] For example, the voice object identifier selected by the user is: Siri voice, and the input text is "The weather is very good today". First, step S111 converts it into word segmentation information: "jin tian tian qi hen hao", and step S112 converts the word segmentation information into a phoneme sequence, expressed as: {j,in,t,i,an,t,i,an,q,i,h,en,h,ao}.

[0095] In one embodiment, the voice object identifier is obtained and in a subsequent processing step, a corresponding voice pronunciation is generated according to the voice object identifier to meet the user's customized voice needs.

[0096] In one embodiment, after obtaining the phoneme sequence and the speech object identifier in step S110, the duration of the phoneme sequence is predicted according to the speech object identifier to obtain a phoneme aligned sequence.

[0097] Step S120 , predicting the duration of the phoneme sequence according to the speech object identifier to obtain a phoneme alignment sequence.

[0098] In one embodiment, the process of obtaining a phoneme alignment sequence is as follows: first, a probability model is trained for each type of phoneme; then, the probability model is used to calculate the posterior probability of the phonemes in the phoneme sequence to obtain a probability matrix; then, dynamic path planning is used to solve the probability matrix for the optimal alignment path; finally, the phoneme position information is mapped to obtain the start and end positions of the phonemes, thereby obtaining the phoneme alignment sequence. This method, when multiple languages ​​correspond to multiple phonemes, requires a large amount of computational effort for phoneme alignment.

[0099] Reference Figure 3 , is a specific implementation flowchart of step S120 shown in one embodiment. In this embodiment, step S120 of predicting the duration of a phoneme sequence based on a speech object identifier to obtain a phoneme alignment sequence includes:

[0100] Step S121: Acquire the speech object embedding and phoneme embedding corresponding to the speech object identifier and the phoneme sequence.

[0101] Step S122: Use a multi-head attention unit to extract attention information from the phoneme embedding to obtain a phoneme representation vector.

[0102] Step S123 , performing duration prediction based on the phoneme representation vector and the speech object embedding to generate the duration of each phoneme in the phoneme sequence.

[0103] Step S124 : obtaining a phoneme alignment sequence according to the phoneme sequence and the duration of each phoneme.

[0104] In one embodiment, the above steps are performed using a pre-trained duration prediction model. The duration prediction model can be trained using training samples to obtain optimal model parameters suitable for the scenario. This embodiment does not specifically limit the training process.

[0105] Embedding is a feature processing method that maps high-dimensional raw data (such as images, sentences, and speech) onto a low-dimensional manifold, making the high-dimensional raw data separable after being mapped onto the low-dimensional manifold. In one embodiment, step S121 embeds the input phoneme sequence and speech object identifier using a pre-trained embedding feature lookup table to obtain corresponding low-dimensional feature vectors: speech object embedding and phoneme embedding. The embedding feature lookup table can be trained through a pre-training process.

[0106] In one embodiment, step S122 uses a multi-head attention unit to extract attention features from the obtained phoneme embedding to obtain a phoneme representation vector. The multi-head attention unit includes multiple attention calculation units. In this embodiment, the multi-head attention unit is a four-layer attention calculation unit. Different attention calculation units include different attention weight sequences. The attention feature vectors of different attention calculation units are calculated in parallel and then concatenated to form the phoneme representation vector.

[0107] In one embodiment, the attention weight sequence of the attention calculation unit includes: query information Q, key K and value V, where the query information Q, key K and value V are all vectors. For example, the query information Q can be the input word segmentation vector W. The attention calculation unit maps the query information Q and the set of key K-value V into an attention feature vector, and multiple attention calculation units include different attention weight sequences, which can obtain different attention feature vectors.

[0108] In one embodiment, the attention weight sequence can be obtained through a learning process, and the calculation process of the attention calculation unit is expressed as:

[0109]

[0110] Among them, d k It represents the dimension of the query information Q, which can be the number of elements of the phoneme embedding. Softmax() represents the Softmax function.

[0111] After obtaining the potential phoneme representation vector with added attention information, step S123 performs duration prediction based on the phoneme representation vector and the speech object embedding to generate the duration of each phoneme in the phoneme sequence.

[0112] Specifically, step S123 first uses two one-dimensional convolutional layers to perform convolution operations on the phoneme representation vector and the speech object embedding respectively, then sums the convolution operation results, and then uses the fully connected layer to generate time information in the logarithmic domain based on the summation result. The time information represents the duration of each phoneme in the phoneme sequence under the speech object identification.

[0113] For example, different speakers have different timbre, which is reflected in their different emphasis choices and pronunciation durations for different words. Consequently, the duration of the same phoneme varies. The above steps target the speech object identifier and obtain the duration of each phoneme in the phoneme sequence associated with that speech object identifier. The phoneme sequence is then aligned at the speech frame level based on the duration of each phoneme to obtain an aligned phoneme sequence.

[0114] For example, in one embodiment:

[0115] Phoneme sequence: {j,in,t,i,an,t,i,an,q,i,h,en,h,ao};

[0116] The duration of each phoneme sequence is expressed as:

[0117] {j-0.06,in-0.09,t-0.03,i-0.03,an-0.06,t-0.03,i-0.03,an-0.06,q-0.03,i-0.09,h-0.03,en-0.06,h-0.03,ao-0.06};

[0118] The duration of a corresponding speech frame is 0.03s. The duration of each phoneme sequence is converted into the number of speech frames, which is expressed as:

[0119] {j-2,in-3,t-1,i-1,an-2,t-1,i-1,an-2,q-1,i-3,h-1,en-2,h-1,ao-2}.

[0120] Reference Figure 4 , which is a schematic diagram of a phoneme alignment sequence. In the figure, a small square represents the duration of a speech frame. After the phoneme sequence is aligned in the above embodiment, the phoneme alignment sequence is aligned according to the duration of each phoneme.

[0121] In one embodiment, a duration prediction model is pre-trained using a large amount of speech training data (including the correspondence between speech objects and phoneme durations). During the training process, the mean square error is used as a loss function, and the mean square error is calculated based on the duration labels in the training data and the predicted duration. The model of the duration prediction model is adjusted according to the preset loss convergence condition, so that the duration prediction model can predict more accurate phoneme durations.

[0122] As can be seen above, the speech object embedding and phoneme embedding corresponding to the speech object identifier and phoneme sequence are obtained. The multi-head attention unit then extracts attention information from the phoneme embedding to obtain a phoneme representation vector. Duration prediction is then performed based on the phoneme representation vector and the speech object embedding to generate the duration of each phoneme in the phoneme sequence. Finally, the phoneme alignment sequence is obtained based on the phoneme sequence and the duration of each phoneme. This approach eliminates the need to train probability models for each type of phoneme in different languages ​​or to use dynamic path planning to solve the probability matrix for the optimal alignment path, thus reducing the computational complexity of the phoneme alignment process.

[0123] In one embodiment, in order to expand the application scenarios of speech synthesis technology and adapt to customized scenarios, the phoneme alignment sequence obtained above is unsupervised mapped to obtain an unsupervised sequence of phonemes, and an unsupervised learning method is introduced to perform speech synthesis on the input text to improve the naturalness and interpretability of the synthesized speech.

[0124] Step S130 : Masking the phoneme alignment sequence based on a preset masking rate and a preset masking step length to obtain a candidate phoneme sequence.

[0125] In one embodiment, the unsupervised mapping process for candidate phoneme sequences involves using a self-organizing map neural network for unsupervised clustering. The input phoneme alignment sequence is classified by finding an optimal set of reference vectors, where each reference vector is a connection weight vector corresponding to an output unit. The resulting cluster centers can be mapped onto a curved or flat surface while maintaining the topological structure, thereby obtaining an unsupervised phoneme sequence. However, the unsupervised phoneme sequence obtained in this manner does not contain additional contextual information.

[0126] Reference Figure 5 , is a flowchart of a specific implementation of step S130 shown in one embodiment. In this embodiment, step S130 of masking the phoneme alignment sequence based on a preset masking rate and a preset masking step length to obtain a candidate phoneme sequence includes:

[0127] Step S131 : obtaining the phoneme alignment embedding of the phoneme alignment sequence using preset alignment lookup table information.

[0128] Step S132: Select the starting position index information.

[0129] Step S133 , masking the phonemes according to the starting position index information, the preset masking rate and the preset masking step length to align and embed the phonemes to obtain a candidate phoneme sequence.

[0130] In one embodiment, embedding is a feature processing method that maps high-dimensional raw data (e.g., images, sentences, speech, etc.) onto a low-dimensional manifold. Step S131 uses preset alignment lookup table information to obtain a phoneme alignment embedding for the phoneme alignment sequence. That is, the phoneme alignment embedding is a low-dimensional feature vector corresponding to the phoneme alignment sequence. The preset alignment lookup table information can be obtained through a pre-training process.

[0131] In one embodiment, the masking prediction training strategy is to iteratively select a certain subset of the phoneme alignment embedding, and the number of elements in the subset reaches a preset masking rate p%. In each iteration, a different starting position can be randomly selected as the starting position index information, and elements with a preset masking step length m are sampled from the phoneme embedding starting from the starting position index information for masking to obtain a candidate phoneme sequence, and the ratio between the masked elements and the total elements in the candidate phoneme sequence is the preset masking rate.

[0132] Reference Figure 6 , which is a schematic diagram of obtaining a candidate phoneme sequence by phoneme alignment embedding. In the figure, the phoneme alignment embedding is represented as {S1, S2, S3, S4, S5, S6, S7, S8, S9, S10}, the preset masking rate p% is 20%, the starting position index information in this iteration is: 3, and the preset masking step size m is: 2. Since the number of elements in the phoneme alignment embedding in one embodiment is 10, according to the preset masking rate, two elements need to be masked, that is, two elements are replaced by a mask. The mask can be 0 or another number. Therefore, according to the masked prediction training strategy, in this iteration, some elements of the phoneme alignment embedding are masked according to the above parameters. The masking mask is represented as "xx", and the candidate phoneme sequence {S1, S2, S3, xx, xx, S6, S7, S8, S9, S10} shown in the figure is obtained.

[0133] Step S140 : performing unsupervised mapping on the phoneme alignment sequence based on the candidate phoneme sequence, and converting the phoneme alignment sequence into a phoneme unsupervised sequence.

[0134] Reference Figure 7 , is a specific implementation flowchart of step S140 shown in one embodiment. In this embodiment, unsupervised mapping is performed on the phoneme alignment sequence based on the candidate phoneme sequence to convert the phoneme alignment sequence into an unsupervised phoneme sequence, step S140, includes:

[0135] Step S141: Obtain phoneme alignment embeddings of the candidate phoneme sequence and the phoneme alignment sequence.

[0136] Step S142 , the candidate phoneme sequence and the phoneme alignment are embedded into the trained bidirectional long short-term memory module for prediction to obtain an unsupervised phoneme sequence.

[0137] In one embodiment, the above steps are performed using a pre-trained alignment mapping model. The alignment mapping model can be trained using training samples to obtain optimal model parameters suitable for the scenario. This embodiment does not specifically limit the training process. In one embodiment, the candidate factor sequence and the phoneme alignment embedding are used as input, and the alignment mapping model is used to predict the unsupervised phoneme sequence corresponding to the phoneme alignment embedding, thereby converting the supervised learning process into an unsupervised learning process.

[0138] In one embodiment, step S142 uses the trained bidirectional long short-term memory module to perform unsupervised prediction on the phoneme embedding to obtain an unsupervised phoneme sequence.

[0139] In one embodiment, the bidirectional long short-term memory module is a Bi-directional Long Short-Term Memory (Bi-LSTM) module, which is composed of a forward LSTM and a backward LSTM, and is used to model the context information corresponding to the phoneme embedding. For example, during the operation process, the forward LSTM sequentially inputs the phoneme embedding {w1, w2, w3} to obtain the corresponding latent vector {h1, h2, h3}, and the backward LSTM sequentially inputs the phoneme embedding {w3, w2, w1} to obtain the corresponding latent vector {h1', h2', h3'}, and the forward and backward latent vectors are spliced ​​together to obtain {[h1, h1'], [h2, h2'], [h3, h3']}, and the corresponding operation is performed. In one embodiment, the bidirectional long short-term memory module is a 3-layer bidirectional long short-term memory module.

[0140] In one embodiment, the alignment mapping model is pre-trained using a large amount of training data. This model's learning process is unsupervised, eliminating the need for time-consuming labeling of the collected data. A masked prediction training strategy is employed during training. This strategy is beneficial for capturing long-term temporal dependencies between different elements, providing more contextual information for each element in the sequence, thereby improving the prediction capabilities of the alignment mapping model.

[0141] In one embodiment, cross entropy is used as a loss function, and the cross entropy loss is calculated over time steps based on the phoneme embeddings before masking in the training data and the candidate phoneme sequences obtained after masking, and is expressed as:

[0142]

[0143]

[0144] L=αLm +(1-α)L u

[0145] Among them, X′ represents the element in the phoneme embedding, L m represents the loss function calculated based on phoneme embedding, z t represents the corresponding element in the predicted unsupervised phoneme sequence, X represents the element in the candidate phoneme sequence, and L u represents the loss function calculated based on the candidate phoneme sequence, L represents the cross-entropy loss value, and α represents the loss weight obtained during the training process. In one embodiment, the persistent bidirectional long short-term memory module is adjusted according to a preset loss convergence condition so that the bidirectional long short-term memory module can predict a more accurate unsupervised phoneme sequence.

[0146] As can be seen from the above, the introduction of unsupervised learning can improve the naturalness and interpretability of synthesized speech by performing unsupervised mapping on phoneme alignment sequences to obtain unsupervised phoneme sequences. This can expand the application scenarios of speech synthesis technology and adapt to customized scenarios.

[0147] In one embodiment, after obtaining the unsupervised phoneme sequence, speech synthesis is performed on the unsupervised phoneme sequence to obtain a spectrogram corresponding to the phoneme sequence, and then the speech corresponding to the speech object identifier is obtained based on the spectrogram.

[0148] Step S150 , performing speech synthesis on the unsupervised phoneme sequence to obtain speech content corresponding to the phoneme sequence.

[0149] In one embodiment, a spectrogram dictionary is constructed using pre-set spectra corresponding to phonemes. The spectrogram corresponding to the phoneme in the unsupervised phoneme sequence is selected from the spectrogram dictionary. The spectrograms of all the phonemes are concatenated to obtain the spectrogram corresponding to the phoneme sequence. The spectrogram is then converted into the corresponding speech content. However, the spectrograms obtained using the spectrogram dictionary cannot meet customization requirements, and the correlation between the spectrogram and the phoneme sequence is too low.

[0150] In one embodiment, the codec model is used to perform step 140, referring to Figure 8 The coding and decoding model includes: a shared encoder 810, a priori encoder 820, a posteriori encoder 830 and a decoder 840, wherein the priori encoder 820 includes: a priori speech object encoder 821 and a priori content encoder 822, and the posteriori encoder 830 includes: a posteriori speech object encoder 831 and a posteriori content encoder 832.

[0151] In one embodiment, referring to Figure 9, is a flowchart of a specific implementation of step S150 shown in an embodiment. In this embodiment, step S150 of performing speech synthesis on an unsupervised phoneme sequence to obtain speech content corresponding to the phoneme sequence includes:

[0152] Step S151: Encode the unsupervised phoneme sequence to obtain a hidden layer feature vector.

[0153] In one embodiment, referring to Figure 8 In the encoding / decoding model, the shared encoder samples the unsupervised sequence of phonemes in the latent variable space to obtain a low-dimensional latent feature vector. In one embodiment, the sampling process is described as follows: the input unsupervised sequence of phonemes is sequentially convolved and pooled, the output of the last convolutional layer is converted into a vector, and this vector is concatenated into a dense layer in the two-dimensional latent space to obtain a low-dimensional latent feature vector.

[0154] Step S152: Probabilistically encode the hidden layer feature vector to obtain speech object distribution and speech content distribution.

[0155] In one embodiment, probability coding includes: posterior probability coding or prior probability coding, the corresponding speech object distribution includes: speech object posterior distribution or speech object prior distribution, and the corresponding speech content distribution includes: speech content posterior distribution or speech content prior distribution.

[0156] refer to Figure 8 In one embodiment, prior information is obtained based on an unsupervised phoneme sequence. The prior information includes prior information about speech objects and prior information about speech content. A posterior speech object encoder is then used to posteriorly encode the hidden feature vectors to obtain a posterior distribution of the speech objects. The prior information about the speech objects is then encoded using a priori speech object encoder to obtain a priori distribution of the speech objects. Furthermore, a posterior speech content encoder is used to posteriorly encode the hidden feature vectors to obtain a posterior distribution of the speech content. The prior information about the speech content is then encoded using a priori speech content encoder to obtain a priori distribution of the speech content.

[0157] In one embodiment, known prior distribution knowledge is used to obtain speech object prior information and speech content prior information corresponding to the unsupervised phoneme sequence. In one embodiment, the hidden layer feature vectors are aligned to a uniform distribution to obtain the corresponding speech object prior information and speech content prior information.

[0158] In one embodiment, the codec model utilizes an unsupervised training process that includes training a posterior encoder and a priori encoder. Specifically, during the training process, the parameters of the posterior encoder and the priori encoder are iteratively adjusted. The posterior encoder encodes its input hidden feature vector to extract features from the hidden feature vector and obtain the corresponding posterior distribution of speech objects and posterior distribution of speech content. These posterior distributions of speech objects and posterior distribution of speech content are the posterior distributions from the input to the output of the posterior encoder. The priori encoder encodes its input prior information to extract features from the prior information and obtain the prior distributions of speech objects and posterior distribution of speech content. During the training process, the parameters of the priori encoder and the posterior encoder are continuously adjusted to ensure that the prior distributions approximate the posterior distributions from the input to the output of the posterior encoder, for example, to ensure that the two are as identical as possible. It should be noted that "as identical as possible" here can mean that the difference between the two is below a certain threshold, that the difference between the two is minimized within a specific interval, or that the difference between the two is minimized within a certain number of iterations during the training process. The specific implementation of this application is not limited to this and can be determined based on actual application requirements.

[0159] In one embodiment, after the training process is completed, the speech object distribution and speech content distribution can be determined based on the trained posterior encoder and / or prior encoder. Since the prior distribution and the posterior distribution are made as similar as possible during the training process, in one embodiment, the speech object distribution and the speech content distribution can be related prior distributions or related posterior distributions, which is not specifically limited in this embodiment.

[0160] In one embodiment, the speech object distribution and speech content distribution may be preset distribution types. For example, if the preset distribution is a normal distribution (also known as a Gaussian distribution), the parameters of the speech object distribution and speech content distribution may include the expectation and standard deviation of the normal distribution, or the parameters of the speech object distribution and speech content distribution may be calculated to obtain the expectation and standard deviation of the normal distribution. Of course, the preset distribution type may also be other types of distributions, as long as their expression can be determined by the a posteriori encoder and / or the a priori encoder.

[0161] After obtaining the speech object distribution and speech content distribution, decoding sampling is required to generate the corresponding spectrogram.

[0162] Step S153: Sampling and encoding the speech object distribution using speech object embedding to obtain a speech object sampling encoding vector.

[0163] Step S154: Sample and encode the speech content distribution using phoneme embedding to obtain a speech content sampling encoding vector.

[0164] In one embodiment, the decoder in the codec model samples the speech object embedding using the parameters of the obtained speech object posterior distribution or speech object prior distribution. The speech object posterior distribution or speech object prior distribution is a probability distribution, and thus the speech object embedding is sampled according to the probability distribution to obtain the speech object sample code vector. Similarly, in one embodiment, the decoder in the codec model uses phoneme embedding to obtain the speech content sample code vector.

[0165] Step S155 , decoding the speech object sampling coding vector and the speech content sampling coding vector to obtain a spectrogram of the phoneme sequence corresponding to the speech object label.

[0166] In one embodiment, the speech object sampling coding vector and the speech content sampling coding vector are spliced, and then the spliced ​​vectors are upsampled. Then, the corresponding spectrogram is selected according to the sampling results for splicing to complete the decoding process and obtain the spectrogram of the phoneme sequence corresponding to the speech object label.

[0167] Step S156: Generate speech content corresponding to the speech object identifier according to the spectrogram.

[0168] In one embodiment, the spectrogram is a mel spectrum. In one embodiment, a vocoder is used to convert the mel spectrum into corresponding speech content. The speech content is a wave file that represents the speech signal in the form of a wave. This embodiment does not limit the form of expression of the speech content.

[0169] As can be seen above, the codec module generates speech object distribution and speech content distribution based on the unsupervised sequence of phonemes in an unsupervised manner, and further obtains the spectrogram of the phoneme sequence corresponding to the speech object label, which can realize customized speech according to speech requirements.

[0170] In one embodiment, a unified framework is used to build a text-to-speech synthesis system, referring to Figure 10 , is a schematic diagram of a text-to-speech synthesis system in one embodiment of the present application. The processing flow is described as follows:

[0171] S1: Obtain input text and speech object identifier.

[0172] S2: Input the input text into the preset phoneme dictionary to obtain the corresponding phoneme sequence.

[0173] S3: Use the duration prediction model to perform frame-level alignment on the phoneme sequence according to the speech object identifier to obtain a phoneme aligned sequence.

[0174] S4: Use the alignment mapping module to perform unsupervised mapping based on the phoneme alignment sequence to obtain an unsupervised phoneme sequence.

[0175] S5: Generate mel-spectrograms from unsupervised sequences of phonemes using the encoder-decoder model.

[0176] S6: Generate a speech waveform based on the Mel-spelling spectrum using a vocoder.

[0177] In one embodiment, the text-to-speech synthesis system connects input text and generated speech, and can realize the generation of customized speech in a more fine-grained manner, thereby expanding the application scenarios of speech synthesis technology.

[0178] The technical solution provided by the embodiment of the present invention obtains the phoneme sequence and speech object identifier corresponding to the input text, masks the phoneme alignment sequence based on a preset masking rate and a preset masking step size to obtain a candidate phoneme sequence, performs unsupervised mapping on the phoneme alignment sequence based on the candidate phoneme sequence, converts the phoneme alignment sequence into an unsupervised phoneme sequence, performs speech synthesis on the unsupervised phoneme sequence, and obtains the speech content corresponding to the phoneme sequence. Compared with the supervised learning process, the technical solution of the present application adopts unsupervised learning, does not require the use of a paired labeled corpus for training, performs speech synthesis on the input text, improves the naturalness and interpretability of the synthesized speech, expands the application scenarios of speech synthesis technology, and can be better applied to application scenarios of customized speech.

[0179] The embodiment of the present invention also provides a text-to-speech synthesis device, which can implement the above-mentioned text-to-speech synthesis method. Figure 11 , the device comprises:

[0180] An acquiring unit 1110 is configured to acquire a phoneme sequence and a speech object identifier corresponding to an input text;

[0181] The duration prediction unit 1120 is configured to predict the duration of the phoneme sequence according to the speech object identifier to obtain a phoneme alignment sequence;

[0182] A sequence masking unit 1130 is configured to mask the phoneme alignment sequence based on a preset masking rate and a preset masking step length to obtain a candidate phoneme sequence;

[0183] An unsupervised mapping unit 1140 is configured to perform unsupervised mapping on the candidate phoneme sequence to obtain an unsupervised phoneme sequence;

[0184] The speech synthesis unit 1150 is used to perform speech synthesis on the unsupervised phoneme sequence to obtain the speech content corresponding to the phoneme sequence.

[0185] The specific implementation of the text-to-speech synthesis apparatus of this embodiment is substantially the same as the specific implementation of the above-mentioned text-to-speech synthesis method, and will not be described in detail here.

[0186] An embodiment of the present invention further provides an electronic device, including:

[0187] at least one memory;

[0188] at least one processor;

[0189] at least one program;

[0190] The program is stored in the memory, and the processor executes the at least one program to implement the text-to-speech synthesis method of the present invention. The electronic device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), an in-vehicle computer, etc.

[0191] See also Figure 12 , Figure 12 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0192] The processor 1201 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0193] The memory 1202 can be implemented in the form of ROM (Read Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1202 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1202 and is called by the processor 1201 to execute the text-to-speech synthesis method of the embodiments of the present invention.

[0194] Input / output interface 1203, used to implement information input and output;

[0195] Communication interface 1204, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0196] Bus 1205 , which transmits information between various components of the device (e.g., processor 1201 , memory 1202 , input / output interface 1203 , and communication interface 1204 );

[0197] The processor 1201 , the memory 1202 , the input / output interface 1203 and the communication interface 1204 are connected to each other in communication within the device via the bus 1205 .

[0198] An embodiment of the present application further provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the above-mentioned text-to-speech synthesis method is implemented.

[0199] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0200] The text-to-speech synthesis method, text-to-speech synthesis device, electronic device, and storage medium proposed in the embodiments of the present invention obtain the phoneme sequence and speech object identifier corresponding to the input text, mask the phoneme alignment sequence based on a preset masking rate and a preset masking step to obtain a candidate phoneme sequence, perform unsupervised mapping on the phoneme alignment sequence based on the candidate phoneme sequence, convert the phoneme alignment sequence into an unsupervised phoneme sequence, perform speech synthesis on the unsupervised phoneme sequence, and obtain the speech content corresponding to the phoneme sequence. Compared with the supervised learning process, the technical solution of the present application adopts unsupervised learning, does not require the use of a paired labeled corpus for training, performs speech synthesis on the input text, improves the naturalness and interpretability of the synthesized speech, expands the application scenarios of speech synthesis technology, and can be better applied to application scenarios of customized speech.

[0201] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0202] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0203] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0204] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0205] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0206] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0207] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0208] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0209] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0210] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0211] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A text-to-speech synthesis method, characterized in that: include: Get the phoneme sequence and speech object identifier corresponding to the input text; Performing duration prediction on the phoneme sequence according to the speech object identifier to obtain a phoneme alignment sequence; Masking the phoneme alignment sequence based on a preset masking rate and a preset masking step length to obtain a candidate phoneme sequence; Performing unsupervised mapping on the phoneme alignment sequence based on the candidate phoneme sequence to convert the phoneme alignment sequence into a phoneme unsupervised sequence; Speech synthesis is performed on the unsupervised phoneme sequence to obtain speech content corresponding to the phoneme sequence.

2. A text-to-speech synthesis method according to claim 1, characterized in that: The step of predicting the duration of the phoneme sequence according to the speech object identifier to obtain a phoneme alignment sequence includes: Obtaining a speech object embedding corresponding to the speech object identifier and a phoneme embedding corresponding to the phoneme sequence; Extracting attention information from the phoneme embedding using a multi-head attention unit to obtain a phoneme representation vector; Performing duration prediction based on the phoneme representation vector and the speech object embedding to generate the duration of each phoneme in the phoneme sequence; The phoneme alignment sequence is obtained according to the phoneme sequence and the duration of each phoneme.

3. A text-to-speech synthesis method according to claim 1, characterized in that: The step of masking the phoneme alignment sequence based on a preset masking rate and a preset masking step length to obtain a candidate phoneme sequence includes: Obtaining a phoneme alignment embedding of the phoneme alignment sequence using preset alignment lookup table information; Select the starting position index information; The candidate phoneme sequence is obtained by masking the phoneme alignment embedding according to the starting position index information, the preset masking rate and the preset masking step size.

4. A text-to-speech synthesis method according to claim 3, characterized in that: The performing unsupervised mapping on the phoneme aligned sequence to obtain an unsupervised phoneme sequence includes: Obtaining the candidate phoneme sequence and the phoneme alignment embedding; The candidate phoneme sequence and the phoneme are aligned and embedded into a trained bidirectional long short-term memory module for prediction to obtain the phoneme unsupervised sequence.

5. A text-to-speech synthesis method according to claim 2, characterized in that: The performing speech synthesis on the unsupervised phoneme sequence to obtain speech content corresponding to the phoneme sequence includes: Performing vector encoding on the unsupervised phoneme sequence to obtain a hidden feature vector; Probabilistically encoding the hidden layer feature vector to obtain speech object distribution and speech content distribution; Sampling and encoding the speech object distribution using the speech object embedding to obtain a speech object sampling encoding vector; Performing sampling encoding on the speech content distribution using the phoneme embedding to obtain a speech content sampling encoding vector; Decoding the speech object sampling coding vector and the speech content sampling coding vector to obtain a spectrogram of the phoneme sequence corresponding to the speech object identifier; The speech content corresponding to the speech object identifier is generated according to the spectrogram.

6. A text-to-speech synthesis method according to claim 5, characterized in that: The probability coding includes posterior probability coding or prior probability coding, the speech object distribution includes: speech object posterior distribution or speech object prior distribution, and the speech content distribution includes: speech content posterior distribution or speech content prior distribution.

7. A text-to-speech synthesis method according to claim 1, characterized in that: The step of obtaining a phoneme sequence corresponding to the input text includes: Converting the input text into word component information; The word segmentation information is mapped into the corresponding phoneme sequence using a preset phoneme dictionary.

8. A text-to-speech synthesis device, characterized in that: include: An acquisition unit, configured to acquire a phoneme sequence and a speech object identifier corresponding to an input text; a duration prediction unit, configured to predict the duration of the phoneme sequence according to the speech object identifier to obtain a phoneme alignment sequence; a sequence masking unit, configured to mask the phoneme alignment sequence based on a preset masking rate and a preset masking step length to obtain a candidate phoneme sequence; an unsupervised mapping unit, configured to perform unsupervised mapping on the candidate phoneme sequence to obtain an unsupervised phoneme sequence; The speech synthesis unit is used to perform speech synthesis on the unsupervised phoneme sequence to obtain speech content corresponding to the phoneme sequence.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the text-to-speech synthesis method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the text-to-speech synthesis method according to any one of claims 1 to 7 is implemented.

Images