Speech synthesis method, speech synthesis device, electronic device, and storage medium

Abstract

The application provides a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, and belongs to the field of financial technology. The method comprises the following steps: obtaining text data, the text data comprising a plurality of characters; performing character feature extraction on the text data to obtain preliminary character features; performing phoneme conversion on the text data to obtain a phoneme sequence; performing phoneme feature extraction on the phoneme sequence to obtain target phoneme features; classifying the characters according to the preliminary character features to obtain category features of the characters; performing feature fusion on the target phoneme features and the category features to obtain fused features; performing prosody feature extraction on the fused features to obtain target prosody features; and performing speech synthesis according to the target prosody features to obtain speech data of the text data, thereby improving the speech quality of the synthesized speech data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of financial technology, and in particular to a speech synthesis method, a speech synthesis device, an electronic device and a storage medium. BACKGROUND

[0002] With the development of artificial intelligence related technology applications, artificial intelligence customer service has been introduced into the banking system to cope with complex financial transaction scenarios, providing intelligent and personalized services for users. In the banking system, artificial intelligence customer service supports complex scenarios such as transfer tasks, account details inquiries, financial tasks, and casual conversations. Artificial intelligence customer service understands and analyzes the input of users through natural language processing technology, identifies the needs and intentions of users, and gives reply texts according to the needs and intentions of users, and converts the reply texts into voice output using speech synthesis technology, which is convenient for users to listen. For example, in the account details inquiry scenario, the user inquires about the historical transaction details of the account through the artificial intelligence customer service, and the artificial intelligence customer service searches the historical transaction records from the database, converts the historical transaction records into voice data, and feeds back the voice data to the user. The historical transaction details involve important information such as transaction amount, transaction party, and transaction type in the past period of time, and need to pay attention to the prosody such as pause, continuity, and tone between words, which has higher requirements for the expressiveness and sound quality of the synthesized voice.

[0003] In related technologies, artificial intelligence customer service converts text data into voice data using an acoustic model including an encoder, a variable adapter, and a decoder to perform speech synthesis. The phoneme sequence is input to the encoder, so that the encoder can learn the context features of the phoneme sequence. However, using only phoneme sequence level features for speech synthesis can make the synthesized voice lack prosody, affecting the voice quality of the synthesized voice data. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose a speech synthesis method, a speech synthesis device, an electronic device and a storage medium, which aims to increase the prosody of the synthesized voice, improve the voice quality of the synthesized voice, and enable the artificial intelligence customer service to provide better experience for users.

[0005] To achieve the above-mentioned purpose, a first aspect of the embodiments of the present application proposes a speech synthesis method, which comprises:

[0006] obtaining text data; the text data comprises a plurality of characters;

[0007] performing character feature extraction on the text data to obtain preliminary character features;

[0008] performing phoneme conversion on the text data to obtain a phoneme sequence;

[0009] perform phoneme feature extraction on the phoneme sequence to obtain target phoneme features;

[0010] classify the character according to the preliminary character features to obtain category features of the character;

[0011] perform feature fusion on the target phoneme features and the category features to obtain fused features;

[0012] perform prosody feature extraction on the fused features to obtain target prosody features;

[0013] perform speech synthesis according to the target prosody features to obtain speech data of the text data.

[0014] In some embodiments, the phoneme feature extraction on the phoneme sequence to obtain target phoneme features includes:

[0015] perform attention feature extraction on the phoneme sequence to obtain attention features;

[0016] perform feature fusion on the phoneme sequence and the attention features to obtain initial phoneme features;

[0017] perform layer normalization processing on the initial phoneme features to obtain first intermediate phoneme features;

[0018] perform feature mapping on the first intermediate phoneme features to obtain second intermediate phoneme features;

[0019] perform feature fusion on the first intermediate phoneme features and the second intermediate phoneme features to obtain third intermediate phoneme features;

[0020] perform layer normalization processing on the third intermediate phoneme features to obtain the target phoneme features.

[0021] In some embodiments, the classification of the character according to the preliminary character features to obtain category features of the character includes:

[0022] perform feature dimension adjustment on the preliminary character features to obtain candidate character features;

[0023] classify the character according to the candidate character features to obtain the category features.

[0024] In some embodiments, the phoneme sequence includes a plurality of phonemes, and the feature fusion on the target phoneme features and the category features to obtain fused features includes:

[0025] obtain a number of the phonemes in the phoneme sequence to obtain a first number;

[0026] obtaining a second quantity of the characters in the text data;

[0027] comparing the first quantity and the second quantity;

[0028] if the first quantity is equal to the second quantity, performing feature fusion on the target phoneme feature and the category feature to obtain a fusion feature.

[0029] In some embodiments, after the comparing the first quantity and the second quantity, the speech synthesis method further comprises:

[0030] if the first quantity is greater than the second quantity, aligning the characters and the phonemes to obtain a third quantity of phonemes corresponding to the characters;

[0031] performing expansion on the category feature according to the third quantity to obtain an expanded feature;

[0032] performing feature fusion on the target phoneme feature and the expanded feature to obtain the fusion feature.

[0033] In some embodiments, the performing prosodic feature extraction on the fusion feature to obtain a target prosodic feature comprises:

[0034] performing phoneme duration feature extraction on the fusion feature to obtain a phoneme duration feature;

[0035] performing length adjustment on the fusion feature according to the phoneme duration feature to obtain a mel-frequency spectrum feature;

[0036] performing pitch feature extraction on the mel-frequency spectrum feature to obtain a pitch feature, performing intensity feature extraction on the mel-frequency spectrum feature to obtain an intensity feature, and performing initial prosodic feature extraction on the mel-frequency spectrum feature to obtain an initial prosodic feature;

[0037] performing feature fusion on the mel-frequency spectrum feature, the pitch feature, the intensity feature, and the initial prosodic feature to obtain the target prosodic feature.

[0038] In some embodiments, the performing speech synthesis according to the target prosodic feature to obtain speech data of the text data comprises:

[0039] performing transpose convolution processing on the target prosodic feature to obtain a first convolution feature;

[0040] performing dilated convolution processing on the first convolution feature to obtain a second convolution feature;

[0041] performing threshold activation processing on the second convolution feature to obtain an activated feature;

[0042] Feature extraction is performed on the activation feature to obtain the speech data.

[0043] To achieve the above object, a second aspect of the embodiment of the present application provides a speech synthesis device, which comprises:

[0044] An acquisition module is configured to acquire text data, wherein the text data comprises a plurality of characters.

[0045] A first feature extraction module is configured to perform character feature extraction on the text data to obtain preliminary character features.

[0046] A phoneme conversion module is configured to perform phoneme conversion on the text data to obtain a phoneme sequence.

[0047] A second feature extraction module is configured to perform phoneme feature extraction on the phoneme sequence to obtain target phoneme features.

[0048] A classification module is configured to classify the characters according to the preliminary character features to obtain category features of the characters.

[0049] A feature fusion module is configured to perform feature fusion on the target phoneme features and the category features to obtain fused features.

[0050] A third feature extraction module is configured to perform prosody feature extraction on the fused features to obtain target prosody features.

[0051] A speech synthesis module is configured to perform speech synthesis according to the target prosody features to obtain speech data of the text data.

[0052] To achieve the above object, a third aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the speech synthesis method of the first aspect when executing the computer program.

[0053] To achieve the above object, a fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the speech synthesis method of the first aspect.

[0054] The voice synthesis method, voice synthesis device, electronic equipment and storage medium provided by the present application obtain text data, perform character feature extraction on the text data to obtain preliminary character features, and increase the prosody of the synthesized voice through the character features. In order to obtain the linguistic features of the text data, phoneme conversion is performed on the text data to obtain a phoneme sequence, phoneme feature extraction is performed on the phoneme sequence to obtain target phoneme features. The characters are classified according to the preliminary character features to obtain category features of the characters, so as to determine the pause of the text data according to the category features, so that the synthesized voice has a sense of rise and fall. Feature fusion is performed on the target phoneme features and the category features to obtain fusion features, so that the fusion features not only include phoneme features but also include character features. Prosody feature extraction is performed on the fusion features to obtain target prosody features, which can enrich the prosody of the synthesized voice. Voice synthesis is performed according to the target prosody features to obtain voice data of the text data, which can increase the prosody of the voice data and improve the effect of voice synthesis. When a user accesses an artificial intelligence customer service through a web customer service, a voice call or the like, the artificial intelligence customer service responds to the query request of the user, and can provide high-quality voice answers matched with the query request in complex scenarios such as a transfer task, account details query, financial management task and casual conversation, so as to provide a better experience for the user. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a flowchart of the voice synthesis method provided by the embodiment of the present application;

[0056] Figure 2 is a flowchart of step S140 in Figure 1

[0057] Figure 3 is a flowchart of step S150 in Figure 1

[0058] Figure 4 is a flowchart of step S160 in Figure 1

[0059] Figure 5 is another flowchart of the voice synthesis method provided by the embodiment of the present application;

[0060] Figure 6 is a flowchart of step S170 in Figure 1

[0061] Figure 7 is a flowchart of step S180 in Figure 1

[0062] Figure 8 is a structural schematic diagram of the voice synthesis device provided by the embodiment of the present application;

[0063] ​​​​​Figure 9 Fig. 1 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0064] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0065] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", and the like in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0067] First, the terms involved in the present application are analyzed:

[0068] Artificial intelligence (AI): is a new technical science to study, develop, simulate, extend and expand human intelligence, and is a branch of computer science. Artificial intelligence aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0069] Self-attention mechanism (Attention Mechanism): Attention mechanism can enable neural networks to focus on a subset of their inputs (or features). Select specific inputs, which can be applied to any type of input regardless of its shape. In the case of limited computing power, attention mechanism is a resource allocation scheme that solves the problem of information overload by allocating computing resources to more important tasks.

[0070] Encoder: transform an input sequence into a fixed-length vector.

[0071] Decoder: transform a previously generated fixed-length vector back into an output sequence; where the input sequence can be text, speech, image, video; the output sequence can be text, image.

[0072] With the development of the application of artificial intelligence related technologies, artificial intelligence customer service has been introduced into the banking system to cope with complex financial transaction scenarios, and to provide intelligent and personalized services for users. In the banking system, the artificial intelligence customer service supports complex scenarios such as transfer tasks, account details inquiries, financial management tasks, and casual conversations. The artificial intelligence customer service understands and analyzes the input of the user through natural language processing technology, identifies the user's needs and intentions, and gives a reply text according to the user's needs and intentions, and converts the reply text into voice output using speech synthesis technology, which is convenient for the user to listen. For example, a user queries the historical transaction details of an account through an artificial intelligence customer service, and the artificial intelligence customer service searches for historical transaction records from the database, converts the historical transaction records into voice data, and feeds back the voice data to the user. The historical transaction details involve important information such as transaction amount, transaction party, and transaction type in the past period of time, and need to pay attention to the rhythm between words, such as pause, continuity, and tone, which has higher requirements for the expressiveness and sound quality of the synthesized voice.

[0073] In related technologies, an acoustic model including an encoder, a variable adapter, and a decoder is used to convert text data into voice data for speech synthesis. The phoneme sequence is input into the encoder, so that the encoder can learn the context features of the phoneme sequence. However, the context features are features between phonemes, and lack word or Chinese character level features. Only using phoneme sequence level features for speech synthesis will make the synthesized voice lack the rhythm between words or characters, such as pause, continuity, and tone, which affects the voice quality of the synthesized voice data.

[0074] Therefore, the embodiments of the present application provide a speech synthesis method, a speech synthesis device, an electronic device, and a computer readable storage medium, which are aimed at increasing the rhythm of the synthesized voice and improving the voice quality of the synthesized voice.

[0075] The speech synthesis method, the speech synthesis device, the electronic device, and the computer readable storage medium provided by the embodiments of the present application are specifically explained through the following embodiments. First, the speech synthesis method in the embodiments of the present application is described.

[0076] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use the knowledge to obtain the best results.

[0077] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0078] The voice synthesis method provided by the embodiments of the present application relates to the field of financial technology. The voice synthesis method provided by the embodiments of the present application can be applied to a terminal, can also be applied to a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application for implementing the voice synthesis method, etc., but is not limited to the above forms.

[0079] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0080] Figure 1 is an optional flowchart of the voice synthesis method provided by the embodiments of the present application, Figure 1The voice synthesis method in the method can include, but is not limited to, steps S110 to S180.

[0081] In step S110, text data is acquired; the text data includes multiple characters.

[0082] In step S120, character feature extraction is performed on the text data to obtain preliminary character features.

[0083] In step S130, phoneme conversion is performed on the text data to obtain a phoneme sequence.

[0084] In step S140, phoneme feature extraction is performed on the phoneme sequence to obtain target phoneme features.

[0085] In step S150, classification is performed on the characters according to the preliminary character features to obtain category features of the characters.

[0086] In step S160, feature fusion is performed on the target phoneme features and the category features to obtain fused features.

[0087] In step S170, prosody feature extraction is performed on the fused features to obtain target prosody features.

[0088] In step S180, voice synthesis is performed according to the target prosody features to obtain voice data of the text data.

[0089] The steps S110 to S180 shown in the embodiments of the present application can obtain preliminary character features by performing character feature extraction on the text data, so as to increase the prosody of the synthesized voice through the character features. In order to obtain the linguistic features of the text data, phoneme conversion is performed on the text data to obtain a phoneme sequence, and target phoneme features are obtained by performing phoneme feature extraction on the phoneme sequence. The category features of the characters are obtained by classifying the characters according to the preliminary character features, so as to determine the pauses of the text data according to the category features, so that the synthesized voice has a sense of rhythm. The fused features are obtained by performing feature fusion on the target phoneme features and the category features, so that the fused features not only include phoneme features but also include character features. The target prosody features are obtained by performing prosody feature extraction on the fused features, which can enrich the prosody of the synthesized voice. The voice data of the text data is obtained by performing voice synthesis according to the target prosody features, which can increase the prosody of the voice data and improve the effect of voice synthesis.

[0090] In step S110 of some embodiments, the user accesses the artificial intelligence customer service through channels such as voice calls, web service, bank applications, etc. The artificial intelligence customer service obtains the question data input by the user, and determines whether the question data is in text form. If the question data is in voice form, the voice form question data is converted into question text, the question text is parsed, the keywords in the question text are recognized, the answer text matching the question text is obtained from the corpus according to the keywords, and the text data is obtained. The artificial intelligence customer service supports complex scenarios such as question and answer, transfer task, account details query, financial task, and casual conversation. The text data can be business text or non-business text unrelated to business questions and casual conversation. The text data includes multiple characters, which are divided into literal characters and punctuation characters. The literal characters can be Chinese characters, English characters, and numerical characters, and the punctuation characters can be punctuation marks.

[0091] For example, in the account details query scenario, the answer text provided by the artificial intelligence customer service is the historical transfer details record of the user from March 2022 to April 2022. The historical transfer details record includes transaction amount, transfer method, and transaction object. If one historical transfer details record is “2022-03-01, transaction amount is 200, transfer method is X application, and transaction object is B”, when the historical transfer details record is converted into voice form, the existing acoustic model encoder can only learn the context features between phoneme sequences, lacks word or Chinese character level features, and causes the synthesized voice data to have no pause between the days and the transactions, incorrect tone of each character, and split between words, etc., which reduces the quality of voice synthesis and affects the user experience.

[0092] For example, in the casual conversation scenario, the user interacts with the artificial intelligence customer service. Since the human-computer interaction is simulated human dialogue, it requires the voice output by the artificial intelligence customer service to be rich in prosody. However, the encoder can only learn phoneme level features, resulting in the voice data output by the artificial intelligence customer service being machine synthesized sound, which lacks emotion and cannot distinguish a complete word or character, reducing the quality of voice synthesis.

[0093] In step S120 of some embodiments, in order to improve the quality of voice synthesis, character features are introduced to enable the artificial intelligence customer service to distinguish complete words or phrases. The text data is subjected to character feature extraction based on a Bidirectional Encoder Representation from Transformers (BERT) model to obtain preliminary character features, which include context semantic features, positional relationship features, etc. If the text data is Chinese text, the preliminary character features are at the level of Chinese characters; if the text data is English text, the preliminary character features are at the level of words. The BERT model is a self-encoding language model that can learn the contextual representation of text data. The BERT model can be any model in the bert-base series, the GPT series, the XLNet series, or the XLM series.

[0094] Taking the bert-base-chinese model as an example, the character feature extraction process for Chinese text data is described. The bert-base-chinese model includes 12 encoding layers, each of which includes a multi-head attention layer, a first normalization layer, a feedforward neural network layer, and a second normalization layer. The input and output of the multi-head attention layer are connected in residual connection, the first normalization layer and the feedforward neural network layer are connected in residual connection, and the feedforward neural network layer is a fully connected mapping network. The multi-head attention layer has 12 heads of attention layers. The Chinese text data is divided by word as the granularity to obtain multiple characters, the characters are mapped to digital indexes based on a dictionary to obtain word embedding vectors, the word embedding vectors are subjected to attention feature extraction by the multi-head attention layer to obtain attention vectors, the word embedding vectors and the attention vectors are fused by vector through residual connection to obtain first feature vectors, the first feature vectors are subjected to layer normalization processing by the first normalization layer to obtain first normalized vectors, the first normalized vectors are subjected to feature mapping by the feedforward neural network layer to obtain second feature vectors, the second feature vectors and the first normalized vectors are fused by vector through residual connection to obtain third feature vectors, the third feature vectors are subjected to layer normalization processing by the second normalization layer to obtain second normalized vectors, the second normalized vectors are taken as the output of the encoder, the output is taken as the input of another encoder, and the output of the 12th encoder is taken as the preliminary character features to obtain the contextual embedding vector representation of each character in the Chinese text data. The pre-trained model bert-base-chinese is selected for character feature extraction, which does not require prior model training, thereby improving the efficiency of feature extraction and the efficiency of voice synthesis.

[0095] In step S130 of some embodiments, linguistic features are extracted from the text data, acoustic features are generated according to the linguistic features, the acoustic features are converted into a speech signal, and speech synthesis is achieved. In order to obtain the linguistic features, the text data is converted into a phoneme sequence by a text phoneme conversion model (Grapheme-to-Phoneme, G2P), and the phoneme sequence includes the phonemes of each character in the text data. If the text data is Chinese text, the phonemes are divided into two parts, the first part is the pinyin of the Chinese character, and the second part is the tone of the Chinese character. The tone includes yin ping, yang ping, shang sheng, and qu sheng. For example, the text data is "nǐ hǎo", and the phonemes of the Chinese character "nǐ" obtained by the G2P model are [nin2], and the phonemes of "hǎo" are [hao3].

[0096] Referring to Figure 2 In some embodiments, step S140 can include but is not limited to steps S210 to S260:

[0097] In step S210, attention feature extraction is performed on the phoneme sequence to obtain attention features.

[0098] In step S220, feature fusion is performed on the phoneme sequence and the attention features to obtain initial phoneme features.

[0099] In step S230, layer normalization processing is performed on the initial phoneme features to obtain first intermediate phoneme features.

[0100] In step S240, feature mapping is performed on the first intermediate phoneme features to obtain second intermediate phoneme features.

[0101] In step S250, feature fusion is performed on the first intermediate phoneme features and the second intermediate phoneme features to obtain third intermediate phoneme features.

[0102] In step S260, layer normalization processing is performed on the third intermediate phoneme features to obtain target phoneme features.

[0103] In step S210 of some embodiments, in order to learn the context features between multiple phonemes in the phoneme sequence, phoneme feature extraction is performed on the phoneme sequence by an encoder to obtain target phoneme features, wherein the encoder includes multiple feed-forward transformer encoding blocks, each of which includes an attention layer and a one-dimensional convolution layer, and the self-attention layer adopts a multi-head attention mechanism. Attention feature extraction is performed on the phoneme sequence by the attention layer to obtain attention features. The output of the previous feed-forward transformer encoding block is taken as the input of the next feed-forward transformer encoding block until the output of the last feed-forward encoding block is obtained, and the output of the last feed-forward encoding block is taken as the target phoneme features. Specifically, the phoneme sequence is represented in a vector form to obtain a phoneme vector, a first weight matrix, a second weight matrix and a third weight matrix are multiplied with the phoneme vector respectively to obtain a query vector corresponding to the first weight matrix, a key vector corresponding to the second weight matrix and a value vector corresponding to the third weight matrix, the attention weight of the value vector is calculated according to the query vector and the key vector, the attention weight is multiplied with the value vector corresponding to the head to obtain the attention vector of each head, and the multiple attention vectors are spliced to obtain the attention features. It should be noted that the query vector is multiplied with the key vector to obtain a first value, the first value is divided by a scaling factor to obtain a second value, the second value is taken as an input parameter of a softmax function to obtain the attention weight.

[0104] In step S220 of some embodiments, as the number of stacked feed-forward transformer encoding blocks increases, the encoder degeneracy phenomenon occurs, resulting in inaccurate extracted target phoneme features. Therefore, the phoneme vector and the attention features are fused to avoid phenomena such as gradient message, gradient explosion, overfitting, etc., to make the forward and backward propagation of phoneme information smoother, and to obtain initial phoneme features.

[0105] In step S230 of some embodiments, in order to improve the efficiency of the encoder in extracting phoneme features, layer normalization processing is performed on the initial phoneme features to obtain first intermediate phoneme features. If the attention layer includes M neurons, the initial phoneme features are composed of vectors output by the M neurons, and the dimension of each vector is l, then the initial phoneme features are represented as The average value μ of the initial phoneme features is obtained by averaging processing as The variance σ of the initial phoneme features is obtained by variance processing as 2 The standard deviation σ of the initial phoneme features is obtained according to the variance as ε is a constant greater than 0, and the initial reference vector is obtained by standardizing the vector according to the average value and the variance as ​​affine transform the initial reference vector according to the scale a and the displacement b to obtain a target reference vector z i For convert the vector of each neuron into the target reference vector to obtain a first intermediate phoneme feature.

[0106] In step S240 of some embodiments, in order to obtain high-level phoneme features, the first intermediate phoneme feature is feature mapped by a one-dimensional convolutional layer to map the first intermediate phoneme feature from a low-dimensional space to a high-dimensional space to obtain a second intermediate phoneme feature.

[0107] In step S250 of some embodiments, in order to alleviate the encoder degradation phenomenon, the first intermediate phoneme feature and the second intermediate phoneme feature are feature added to obtain a third intermediate phoneme feature.

[0108] In step S260 of some embodiments, in order to improve the efficiency of the encoder in extracting phoneme features, the third intermediate phoneme feature is layer normalized to obtain a target phoneme feature, and the method of layer normalization is the same as that in step S230, which will not be repeated here.

[0109] Through the above steps S210 to S260, the context features of the phoneme sequence can be extracted to obtain the phoneme features, and the acoustic features can be obtained based on the phoneme features, and the speech synthesis can be performed according to the acoustic features.

[0110] Please refer to Figure 3 In some embodiments, step S150 can include but is not limited to steps S310 to S320:

[0111] Step S310, adjusting the feature dimension of the preliminary character feature to obtain a candidate character feature;

[0112] Step S320, classifying the character according to the candidate character feature to obtain a category feature.

[0113] In step S310 of some embodiments, the preliminary character feature is input into a fully connected mapping network for character classification to obtain a category feature of the character, wherein the fully connected mapping network includes a linear layer and a classification layer, the feature dimension of the preliminary character feature extracted by the BERT model is 768, and the feature dimension of the target phoneme feature extracted by the encoder is 256. Since the feature dimensions of the preliminary character feature and the target phoneme feature are inconsistent, the category feature cannot be feature fused with the target phoneme feature, therefore, the linear layer is used to feature map the initial character feature to map 768 to 256 to obtain a candidate character feature, so that the feature dimension of the candidate character feature is the same as that of the target phoneme feature.

[0114] In step S320 of some embodiments, the classification layer is configured to perform a three-class classification, and the class feature is configured to identify the class of the character. If the text data is Chinese text data, the candidate character feature is input into the classification layer, and the character is classified according to the candidate character feature, so as to obtain one of the three classes, i.e., a Hanzi character at the beginning of a sentence, a Hanzi character at a position other than the beginning of a sentence, or a punctuation mark.

[0115] It should be noted that the full connection mapping network can be trained in the following manner. Sample Chinese data is obtained, the sample Chinese data includes a plurality of characters, the characters have reference class labels, character features of the sample Chinese data are extracted, the sample character features are input into the full connection mapping network to perform class prediction, and predicted class labels of the characters are obtained. The difference between the reference class labels and the predicted class labels is evaluated by a loss function to obtain loss data, and the network parameters of the full connection mapping network are adjusted according to the loss data to train the full connection mapping network. If the character is a Hanzi character at the beginning of a sentence, the reference class label is 0, if the character is a Hanzi character at a position other than the beginning of a sentence, the reference class label is 1, and if the character is a punctuation mark, the reference class label is 2.

[0116] The steps S310 to S320 described above can solve the problem of inconsistent feature dimensions between the preliminary character features and the target phoneme features by adjusting the feature dimensions of the preliminary character features, and can obtain the class of each character in the text data by classifying the characters, so that the continuity or pause of the text data can be determined based on the class, so that the synthesized speech has a feeling of ups and downs.

[0117] Please refer to Figure 4 In some embodiments, the phoneme sequence includes a plurality of phonemes, and step S160 can include but is not limited to steps S410 to S440:

[0118] In step S410, the number of phonemes in the phoneme sequence is obtained to obtain a first number;

[0119] In step S420, the number of characters in the text data is obtained to obtain a second number;

[0120] In step S430, the first number and the second number are compared;

[0121] In step S440, if the first number and the second number are equal, the feature fusion is performed on the target phoneme feature and the class feature to obtain the fused feature.

[0122] In step S410 of some embodiments, when the feature dimensions between the target phoneme features and the category features are consistent, there may be a problem that the number of phonemes and the number of characters are inconsistent, which causes the target phoneme features and the category features to be unable to be fused, and therefore it is necessary to determine whether the number of phonemes is equal to the number of characters. The number of phonemes in the phoneme sequence is obtained, and a first number is obtained. For example, the phoneme sequence is [[nin2], [hao3]], the number of phonemes of the first element in the sequence is 2, the number of phonemes of the second element is 2, and the first number is 4.

[0123] In step S420 of some embodiments, the text data is “nǐ hǎo”, the number of characters in the text data is obtained, the text data includes two Chinese characters, and a second number is obtained as 2.

[0124] In step S430 of some embodiments, in order to determine whether the first number and the second number are equal, the first number and the second number are compared.

[0125] In step S440 of some embodiments, if the first number and the second number are equal, it indicates that the number of characters of the text data and the number of phonemes of the phoneme sequence are equal, and the category features and the target phoneme features can be fused. The target phoneme features and the category features are fused to obtain fused features.

[0126] The steps S410 to S440 described above can make the category features and the target phoneme features be fused by comparing the number of characters and the number of phonemes, so that the fused features contain both phoneme features and character features, and the diversity of the features is increased to improve the quality of the synthesized speech.

[0127] Please refer to Figure 5 In some embodiments, after step S430, the speech synthesis method can further include but is not limited to steps S510 to S530:

[0128] In step S510, if the first number is greater than the second number, the characters and the phonemes are aligned to obtain a third number of characters corresponding to phonemes;

[0129] In step S520, the category features are expanded according to the third number to obtain expanded features;

[0130] In step S530, the target phoneme features and the expanded features are fused to obtain fused features.

[0131] In step S510 of some embodiments, if the number of phonemes is greater than the number of characters, indicating that the category feature and the target phoneme feature cannot be fused, the characters and the phonemes are aligned to obtain a mapping relationship between the characters and the phonemes, the number of phonemes corresponding to a character is obtained according to the mapping relationship, and a third number is obtained. For example, the phoneme sequence is [[nin2], [hao3]], the text data is “You are welcome”, the first number is 4, the second number is 2, the first number is greater than the second number, the characters of the text data and the phonemes of the phoneme sequence are aligned, and the mapping relationship is {“You”: “nin2”, “good”: “hao3”}. According to the mapping relationship, the third number of the phoneme “nin2” corresponding to the character “You” is 2, and the third number of the phoneme “hao3” corresponding to the character “good” is 2.

[0132] In step S520 of some embodiments, one character has one category feature, one character has multiple phonemes, and each phoneme has one phoneme feature. Therefore, one character has multiple phoneme features, and the category features of each character constitute the category features of the text data, resulting in inconsistent numbers of category features and phoneme features. In order to make the category features of the characters equal in number to the phoneme features corresponding to the characters, the category features are expanded according to the third number, from one to the third number, to obtain expanded features.

[0133] In step S530 of some embodiments, the expanded features of each character constitute the expanded features of the text data, and the expanded features of the text data are added to the target phoneme features to obtain fused features.

[0134] The above steps S510 to S530 can solve the problem of inconsistent numbers of category features and phoneme features, enable the category features of the text data and the target phoneme features of the phoneme sequence to be fused, and perform speech synthesis based on the fused features.

[0135] Please refer to Figure 6 In some embodiments, step S170 can include but is not limited to steps S610 to S640:

[0136] In step S610, phoneme duration feature extraction is performed on the fused features to obtain phoneme duration features.

[0137] In step S620, the length of the fused features is adjusted according to the phoneme duration features to obtain mel-frequency spectrum features.

[0138] In step S630, pitch feature extraction is performed on the mel-frequency spectrum features to obtain pitch features, intensity feature extraction is performed on the mel-frequency spectrum features to obtain intensity features, and initial prosody feature extraction is performed on the mel-frequency spectrum features to obtain initial prosody features.

[0139] In step S640, the mel-spectrum features, the pitch features, the intensity features and the initial prosody features are fused to obtain target prosody features.

[0140] In step S610 of some embodiments, the fused features are subjected to prosody feature extraction by a variable adapter to obtain target prosody features, so as to convert linguistic features into acoustic features. The variable adapter includes a phoneme duration predictor, a length adjuster, a pitch predictor, an intensity predictor and an initial prosody predictor. The network structures of the phoneme duration predictor, the pitch predictor, the intensity predictor and the initial prosody predictor are the same, and each includes a first convolutional layer, a second convolutional layer and a linear layer. The first convolutional layer and the second convolutional layer each adopt a 1D convolutional neural network. The phoneme duration predictor is used to extract phoneme duration features from the fused features, and the phoneme duration features are used to represent phoneme duration time, which can be used to control the speech rate of synthesized speech. The phoneme duration time is the length of the mel-spectrum sequence corresponding to the phoneme. The first convolutional layer is used to perform convolutional operation on the fused features to obtain a convolutional vector. The convolutional vector is subjected to RuLU activation processing to obtain an activation vector. The activation vector is subjected to layer normalization processing to obtain a normalized vector. The normalized vector is subjected to random inactivation processing to obtain a first duration vector. The second convolutional layer is used to perform convolutional operation on the first duration vector to obtain a second duration vector. The linear layer is used to perform linear mapping on the second duration vector to obtain the phoneme duration features. It should be noted that the processing procedures of the first convolutional layer and the second convolutional layer are the same, and will not be described here.

[0141] In step S620 of some embodiments, one phoneme corresponds to multiple mel-spectrum sequences, and the phoneme and the mel-spectrum sequence have length difference. In order to adjust the length difference and control the sound speed of synthesized speech, the phoneme duration features and the fused features are input into the length adjuster. The fused features are tiled according to the phoneme duration time to match the length of the mel-spectrum sequence corresponding to the fused features, so as to obtain mel-spectrum features. By lengthening or shortening the phoneme duration time, the sound speed of synthesized speech can be controlled. In addition, by feature alignment, the phenomenon of repeated reading and missed reading of synthesized speech can be avoided.

[0142] In step S630 of some embodiments, the mel-spectral features are pitch predicted by a pitch predictor to obtain a pitch spectrum, the pitch spectrum is inverse continuous wavelet transformed to convert the pitch spectrum into a fundamental curve, and a plurality of fundamental frequencies in the fundamental curve are taken as pitch features, which are used to represent the high or low of the fundamental frequency of the sound. The mel-spectral features are intensity predicted by an intensity predictor to obtain intensity features, which are used to represent the intensity of the sound. The mel-spectral features are initial prosody features extracted by an initial prosody predictor to obtain initial prosody features, which are used to represent the content of the speech, i.e., the pronunciation of each character. The process of pitch prediction by the pitch predictor, the process of intensity prediction by the intensity predictor, and the process of initial prosody prediction by the initial prosody predictor are the same as the process of phoneme duration prediction by the phoneme duration predictor, and will not be repeated here.

[0143] It should be noted that the training process of the phoneme duration predictor is as follows: the phoneme sequence of the sample text and the sample audio of the sample text are obtained, the phoneme sequence includes a plurality of phonemes, and the audio sequence includes a plurality of mel frames. The phonemes and the mel frames are aligned by an alignment tool to obtain the frame number of the mel frame corresponding to each phoneme, and the frame number is taken as the reference phoneme duration. The phoneme sequence is phoneme duration predicted to obtain predicted phoneme duration. The difference between the reference phoneme duration and the predicted phoneme duration is evaluated by a mean square error loss function to obtain loss data. The network parameters of the model are adjusted according to the loss data to train the model to obtain the phoneme duration predictor.

[0144] The training process of the pitch predictor includes: decomposing the sample pitch sequence into a reference pitch spectrum by continuous wavelet transform, and predicting the pitch spectrum of the phoneme sequence to obtain a predicted pitch spectrum. The difference between the reference pitch spectrum and the predicted pitch spectrum is evaluated by a mean square error loss function to obtain loss data. The network parameters of the model are adjusted according to the loss data to train the model to obtain the pitch predictor.

[0145] The training process of the intensity predictor includes: converting the sample audio from the time domain to the frequency domain by short-time Fourier transform to obtain a short-time spectrum, the short-time spectrum includes a short-time amplitude spectrum and a short-time phase spectrum, and the short-time amplitude spectrum includes a plurality of short-time amplitudes. The L2 norm of the short-time amplitude is taken as the reference intensity, the phoneme sequence is intensity predicted to obtain predicted intensity. The difference between the predicted intensity and the reference intensity is evaluated by a mean square error loss function to obtain loss data. The network parameters of the model are adjusted according to the loss data to train the model to obtain the intensity predictor.

[0146] The training process of the initial prosody predictor includes: performing pronunciation prediction on the phoneme sequence to obtain predicted audio for each character. The difference between the sample audio and the predicted audio is evaluated by a mean square error loss function to obtain loss data. The network parameters of the model are adjusted according to the loss data to train the model to obtain the initial prosody predictor.

[0147] In step S640 of some embodiments, the mel-spectral features, pitch features, intensity features, and initial prosody features are subjected to feature summation processing to enhance the contextual prosody features and enhance the prosody effect of the synthesized speech to obtain target prosody features.

[0148] The steps S610 to S640 described above can enrich the prosody features by extracting phoneme duration features, pitch features, intensity features, and initial prosody features to improve the speech quality of the synthesized speech.

[0149] Please refer to Figure 7 In some embodiments, step S180 can include but is not limited to steps S710 to S740:

[0150] Step S710, transposed convolution processing is performed on the target prosody features to obtain first convolution features;

[0151] Step S720, the first convolution features are subjected to dilated convolution processing to obtain second convolution features;

[0152] Step S730, threshold activation processing is performed on the second convolution features to obtain activation features;

[0153] Step S740, feature extraction is performed on the activation features to obtain speech data.

[0154] In step S710 of some embodiments, the target prosody features are input to a decoder, speech synthesis is performed according to the target prosody features, and text data is directly converted into speech signals to obtain speech data of the text data. The decoder includes a plurality of decoding blocks and a 1D convolution layer. The decoding block includes a 1D transposed convolution layer, a 1D dilated convolution layer, a threshold activation layer, and a 1x1 convolution layer. The target prosody features are subjected to transposed convolution by the 1D transposed convolution layer to map the target prosody features from low-dimensional features to high-dimensional features to improve the resolution of the speech and obtain first convolution features.

[0155] In step S720 of some embodiments, the first convolution features are subjected to dilated convolution by the 1D dilated convolution layer to increase the receptive field of the convolution kernel and avoid losing a large amount of local feature information to obtain second convolution features.

[0156] In step S730 of some embodiments, the second convolutional features are threshold activated by a threshold activation layer to determine features input to a next decoding block. If an excitation value of a current neuron exceeds a threshold value, the neuron is activated and features input to the neuron can be transmitted to the next decoding block to obtain activated features.

[0157] In step S740 of some embodiments, the output of a previous decoding block is taken as input of a current decoding block to obtain the activated features. The activated features output by the last decoding block are input to a 1x1 convolutional layer for dimension adjustment, and the dimension-adjusted features are input to a 1D convolutional layer for feature extraction to obtain the speech data.

[0158] The steps S710 to S740 described above convert text data into speech information by waveform decoding of target prosodic features, obtain speech data, and improve prosodic effects of the speech data, make the speech data have emotional characteristics, and enhance user experience.

[0159] Please refer to Figure 8 The embodiments of the present disclosure also provide a speech synthesis device, which can implement the speech synthesis method described above. The speech synthesis device comprises:

[0160] The acquisition module 810 is configured to acquire text data. The text data comprises a plurality of characters.

[0161] The first feature extraction module 820 is configured to perform character feature extraction on the text data to obtain preliminary character features.

[0162] The phoneme conversion module 830 is configured to perform phoneme conversion on the text data to obtain a phoneme sequence.

[0163] The second feature extraction module 840 is configured to perform phoneme feature extraction on the phoneme sequence to obtain target phoneme features.

[0164] The classification module 850 is configured to classify the characters according to the preliminary character features to obtain category features of the characters.

[0165] The feature fusion module 860 is configured to perform feature fusion on the target phoneme features and the category features to obtain fused features.

[0166] The third feature extraction module 870 is configured to perform prosodic feature extraction on the fused features to obtain target prosodic features.

[0167] The speech synthesis module 880 is configured to perform speech synthesis according to the target prosodic features to obtain speech data of the text data.

[0168] The specific implementation of the speech synthesis device is basically the same as that of the specific embodiments of the speech synthesis method described above, and will not be repeated here.

[0169] The embodiment of the present application further provides an electronic device, which comprises a memory and a processor. The memory stores a computer program. The processor executes the computer program to realize the voice synthesis method. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0170] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated, which comprises:

[0171] The processor 910 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is used to execute related programs to realize the technical solutions provided by the embodiments of the present application.

[0172] The memory 920 can be implemented in the form of a ROM (ReadOnly Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory). The memory 920 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 920 and are called and executed by the processor 910 to realize the voice synthesis method of the embodiments of the present application.

[0173] The input / output interface 930 is used to realize information input and output.

[0174] The communication interface 940 is used to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0175] The bus 950 is used to transmit information between various components (for example, the processor 910, the memory 920, the input / output interface 930, and the communication interface 940) of the device.

[0176] The processor 910, the memory 920, the input / output interface 930, and the communication interface 940 are connected to each other through the bus 950 to realize the communication connection between them in the device.

[0177] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to realize the voice synthesis method.

[0178] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory that is remotely arranged with respect to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0179] The voice synthesis method, the voice synthesis device, the electronic equipment and the computer readable storage medium provided by the embodiments of the present application can obtain the preliminary character features by performing character feature extraction on the text data, so as to increase the prosody of the synthesized voice through the character features. In order to obtain the linguistic features of the text data, the text data is subjected to phoneme conversion to obtain a phoneme sequence, and the phoneme sequence is subjected to phoneme feature extraction to obtain target phoneme features. The characters are classified according to the preliminary character features to obtain the category features of the characters, so as to determine the pause of the text data according to the category features, so that the synthesized voice has a sense of rise and fall. The target phoneme features and the category features are subjected to feature fusion to obtain fusion features, so that the fusion features not only include phoneme features but also include character features. The fusion features are subjected to prosody feature extraction to obtain target prosody features, which can enrich the prosody of the synthesized voice. The voice synthesis is performed according to the target prosody features to obtain voice data of the text data, which can increase the prosody of the voice data and improve the effect of voice synthesis.

[0180] The embodiments described in the embodiments of the present application are used to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0181] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than the figures shown, or combine certain steps, or different steps.

[0182] The device embodiments described above are only schematic, and the units illustrated as separate components can or can not be physically separate, that is, can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0183] Those skilled in the art can understand that all or some of the steps in the method disclosed above, the function modules / units in the system and the device can be implemented as software, firmware, hardware or appropriate combination thereof.

[0184] The terms "first", "second", "third", "fourth" and the like in the description of the application and in the claims, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of these terms herein is to be construed to cover a general order and / or structure unless otherwise indicated. Furthermore, the terms "comprise", "comprising", "has", "having", "includes", "including", "contain", "containing" or any other similar forms are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises, has, includes, contains items or components does not include items or components not explicitly recited. The terms "a" or "an", as used herein in the detailed description and in the claims, mean "one or more" or "at least one", unless otherwise indicated.

[0185] It should be understood that, in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the relationship between associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can mean a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be singular or plural.

[0186] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed objects can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0187] The units described as separate components above can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place, or can be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0188] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0189] If the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0190] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A speech synthesis method characterized by, The method comprises: acquiring text data; the text data comprises a plurality of characters; character feature extraction is performed on the text data to obtain preliminary character features; phoneme conversion is performed on the text data to obtain a phoneme sequence; phoneme feature extraction is performed on the phoneme sequence to obtain target phoneme features; the characters are classified according to the preliminary character features to obtain category features of the characters; feature fusion is performed on the target phoneme features and the category features to obtain fused features; prosody feature extraction is performed on the fused features to obtain target prosody features; speech synthesis is performed according to the target prosody features to obtain speech data of the text data. The classification of the characters according to the preliminary character features to obtain the category features of the characters comprises: feature dimension adjustment is performed on the preliminary character features to obtain candidate character features; the characters are classified according to the candidate character features to obtain the category features.

2. The speech synthesis method of claim 1, wherein, The phoneme feature extraction on the phoneme sequence to obtain the target phoneme features comprises: attention feature extraction is performed on the phoneme sequence to obtain attention features; feature fusion is performed on the phoneme sequence and the attention features to obtain initial phoneme features; layer normalization processing is performed on the initial phoneme features to obtain first intermediate phoneme features; feature mapping is performed on the first intermediate phoneme features to obtain second intermediate phoneme features; feature fusion is performed on the first intermediate phoneme features and the second intermediate phoneme features to obtain third intermediate phoneme features; layer normalization processing is performed on the third intermediate phoneme features to obtain the target phoneme features.

3. The speech synthesis method of claim 1, wherein, The phoneme sequence comprises a plurality of phonemes, and the feature fusion on the target phoneme features and the category features to obtain the fused features comprises: a number of the phonemes in the phoneme sequence is acquired to obtain a first number; a number of the characters in the text data is acquired to obtain a second number; the first number and the second number are compared; if the first number is equal to the second number, feature fusion is performed on the target phoneme features and the category features to obtain the fused features.

4. The speech synthesis method according to claim 3, characterized by, After the comparison of the first number and the second number, the speech synthesis method further comprises: if the first number is greater than the second number, the characters and the phonemes are aligned to obtain a third number of phonemes corresponding to the characters; the category features are expanded according to the third number to obtain expanded features; feature fusion is performed on the target phoneme features and the expanded features to obtain the fused features.

5. The speech synthesis method according to any one of claims 1 to 4, characterized by, The prosody feature extraction on the fused features to obtain the target prosody features comprises: phoneme duration feature extraction is performed on the fused features to obtain phoneme duration features; length adjustment is performed on the fused features according to the phoneme duration features to obtain mel-frequency spectrum features; pitch feature extraction is performed on the mel-frequency spectrum features to obtain pitch features, intensity feature extraction is performed on the mel-frequency spectrum features to obtain intensity features, and initial prosody feature extraction is performed on the mel-frequency spectrum features to obtain initial prosody features; The mel-frequency spectrum feature, the pitch feature, the intensity feature, and the initial prosody feature are fused to obtain the target prosody feature.

6. The speech synthesis method according to any one of claims 1 to 4, characterized by, The speech synthesis according to the target prosody feature obtains speech data of the text data, and the speech synthesis according to the target prosody feature includes: The target prosody feature is subjected to transposed convolution processing to obtain a first convolution feature; The first convolution feature is subjected to dilated convolution processing to obtain a second convolution feature; The second convolution feature is subjected to threshold activation processing to obtain an activated feature; The activated feature is subjected to feature extraction to obtain the speech data.

7. A speech synthesis apparatus characterized by comprising: The device includes: An acquisition module is configured to acquire text data, and the text data includes a plurality of characters; A first feature extraction module is configured to perform character feature extraction on the text data to obtain preliminary character features; A phoneme conversion module is configured to perform phoneme conversion on the text data to obtain a phoneme sequence; A second feature extraction module is configured to perform phoneme feature extraction on the phoneme sequence to obtain target phoneme features; A classification module is configured to classify the characters according to the preliminary character features to obtain category features of the characters; A feature fusion module is configured to fuse the target phoneme features and the category features to obtain fused features; A third feature extraction module is configured to perform prosody feature extraction on the fused features to obtain target prosody features; A speech synthesis module is configured to perform speech synthesis according to the target prosody features to obtain speech data of the text data; The speech synthesis device is further configured to: Perform feature dimension adjustment on the preliminary character features to obtain candidate character features; Classify the characters according to the candidate character features to obtain the category features.

8. An electronic device, characterized by The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the speech synthesis method in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to implement the speech synthesis method in any one of claims 1 to 6.

Images