Speech synthesis method, device, computer equipment and storage medium

Abstract

The present application relates to a speech synthesis method, device, computer equipment and storage medium. The method comprises: determining the coding segment of the sound spectrum to be predicted by encoding the phoneme feature information corresponding to the Chinese text; in each round of iterative prediction, determining the current sound spectrum feature corresponding to the coding segment predicted in this round, and the attention weight used when predicting the current sound spectrum feature in this round; when the attention weight reaches a preset weight threshold, adjusting the preset number of decoding times corresponding to the coding segment based on the length of the current sound spectrum feature to obtain the adjusted preset number of decoding times; when the number of iterative prediction rounds reaches the preset number of decoding times, stopping the iteration, and determining the current sound spectrum feature as the target sound spectrum feature corresponding to the coding segment; the target sound spectrum feature is used to synthesize the speech corresponding to the Chinese text. The use of this method can improve the accuracy of speech synthesis.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech synthesis method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of speech processing technology, speech synthesis technology has emerged. Speech synthesis technology can transform text information generated by the computer itself or input from the outside into understandable and fluent Chinese spoken output, thereby realizing voice communication of computer equipment.

[0003] Traditional autoregressive-based speech synthesis models require a large amount of training data. If the training data is not enough, the speech synthesis model may easily end decoding prematurely or fail to end when performing spectrogram prediction, resulting in spectrogram prediction errors, which in turn leads to inaccurate speech synthesis. Summary of the invention

[0004] Based on this, it is necessary to provide a speech synthesis method, apparatus, computer device, computer-readable storage medium and computer program product that can improve accuracy in order to solve the above technical problems.

[0005] In a first aspect, the present application provides a speech synthesis method. The method comprises:

[0006] By encoding the phoneme feature information corresponding to the Chinese text, the encoding segment of the sound spectrum to be predicted is determined;

[0007] In each round of iterative prediction, determining the current spectral feature corresponding to the encoding segment predicted in this round, and the attention weight used when predicting the current spectral feature in this round;

[0008] When the attention weight reaches a preset weight threshold, adjusting the preset number of decoding times corresponding to the coding segment based on the length of the current sound spectrum feature to obtain an adjusted preset number of decoding times;

[0009] When the number of iterative prediction rounds reaches the preset number of decoding times, the iteration is stopped, and the current spectral feature is determined as the target spectral feature corresponding to the encoding segment; the target spectral feature is used to synthesize the speech corresponding to the Chinese text.

[0010] In one embodiment, the step of encoding the phoneme feature information corresponding to the Chinese text to determine the encoding segment of the sound spectrum to be predicted comprises:

[0011] Determine phoneme feature information corresponding to the Chinese text;

[0012] Encoding the phoneme feature information to obtain a text encoding sequence corresponding to the Chinese text;

[0013] According to the preset time step, the coding segment to be subjected to the spectrogram prediction is determined from the text coding sequence.

[0014] In one embodiment, the method further comprises:

[0015] When the number of iterative prediction rounds does not reach the preset number of decoding times, the next round is taken as the current round, and the step of returning to determine the current spectral features corresponding to the coding segment predicted in the current round, and the attention weight used when predicting the current spectral features in the current round is continued.

[0016] In one embodiment, the method further comprises a step of calculating an attention weight; the step of calculating an attention weight comprises:

[0017] In each round of iterative prediction, determine the current attention vector used when predicting the current spectral features in this round;

[0018] Based on the similarity between the current attention vector and the encoding segment, an attention weight corresponding to the current attention vector is calculated; the attention weight is used to indicate feature extraction of the attention vector to obtain a current feature vector; the current feature vector is used to predict current spectrum features.

[0019] In one embodiment, when the attention weight reaches a preset weight threshold, the preset number of decoding times corresponding to the encoding segment is adjusted based on the length of the current spectrum feature, and the adjusted preset number of decoding times includes:

[0020] When the attention weight reaches a preset weight threshold, determining a decoding round offset value corresponding to the encoding segment;

[0021] According to the decoding round offset value and the length of the current sound spectrum feature, the preset number of decoding times corresponding to the coding segment is adjusted to obtain an adjusted preset number of decoding times.

[0022] In one embodiment, the method further comprises:

[0023] Predicting end identification information corresponding to the encoding segment based on the current sound spectrum feature;

[0024] When the end identification information cannot meet the preset end condition, the number of iterative prediction rounds is compared with the preset number of decoding times, and the step of stopping iteration is performed when the number of iterative prediction rounds matches the preset number of decoding times.

[0025] In one embodiment, the method further comprises:

[0026] When the end identification information satisfies a preset end condition, the iteration is stopped, and the current sound spectrum feature is determined as the target sound spectrum feature corresponding to the encoding segment.

[0027] In a second aspect, the present application also provides a speech synthesis device. The device comprises:

[0028] An encoding module, used to determine the encoding segment of the sound spectrum to be predicted by encoding the phoneme feature information corresponding to the Chinese text;

[0029] A decoding module, in each round of iterative prediction, determines the current spectral feature corresponding to the coding segment predicted in this round, and the attention weight used in predicting the current spectral feature in this round; when the attention weight reaches a preset weight threshold, adjusts the preset number of decoding times corresponding to the coding segment based on the length of the current spectral feature to obtain the adjusted preset number of decoding times; when the number of iterative prediction rounds reaches the preset number of decoding times, stops iteration, and determines the current spectral feature as the target spectral feature corresponding to the coding segment; the target spectral feature is used to synthesize the speech corresponding to the Chinese text.

[0030] In one of the embodiments, the encoding module is further used to determine the phoneme feature information corresponding to the Chinese text; encode the phoneme feature information to obtain a text encoding sequence corresponding to the Chinese text; and determine the encoding segment to be subjected to spectrogram prediction from the text encoding sequence according to a preset time step.

[0031] In one embodiment, the decoding module is further used to, when the number of iterative prediction rounds does not reach the preset decoding number, take the next round as the current round, return to the step of determining the current spectral features corresponding to the coding segment predicted in the current round, and the attention weight used when predicting the current spectral features in the current round, and continue to execute.

[0032] In one of the embodiments, the decoding module is also used to determine, in each round of iterative prediction, a current attention vector used when predicting the current spectral features of this round; based on the similarity between the current attention vector and the encoding segment, calculate the attention weight corresponding to the current attention vector; the attention weight is used to indicate feature extraction of the attention vector to obtain the current feature vector; the current feature vector is used to predict the current spectral features.

[0033] In one of the embodiments, the decoding module is further used to determine a decoding round offset value corresponding to the coding segment when the attention weight reaches a preset weight threshold; and adjust the preset number of decoding times corresponding to the coding segment according to the decoding round offset value and the length of the current spectral feature to obtain an adjusted preset number of decoding times.

[0034] In one of the embodiments, the decoding module is also used to predict the end identification information corresponding to the encoding segment based on the current sound spectrum features; when the end identification information cannot meet the preset end condition, the number of iterative prediction rounds is compared with the preset number of decoding times, and the step of stopping the iteration is executed when the number of iterative prediction rounds matches the preset number of decoding times.

[0035] In one of the embodiments, the decoding module is further used to stop iteration when the end identification information meets a preset end condition, and determine the current sound spectrum feature as the target sound spectrum feature corresponding to the encoding segment.

[0036] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in each embodiment of the method described in the present application are implemented.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in each embodiment of the method described in the present application are implemented.

[0038] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps in each embodiment of the method described in the present application are implemented.

[0039] The speech synthesis method, device, computer equipment, storage medium and computer program product described above, by encoding the phoneme feature information corresponding to the Chinese text, determine the coding segment of the sound spectrum to be predicted; in each round of iterative prediction, determine the current sound spectrum feature corresponding to the coding segment predicted in this round, and the attention weight used when predicting the current sound spectrum feature in this round; when the attention weight reaches the preset weight threshold, adjust the preset number of decoding times corresponding to the coding segment based on the length of the current sound spectrum feature to obtain the adjusted preset number of decoding times; when the number of iterative prediction rounds reaches the preset number of decoding times, stop the iteration, and determine the current sound spectrum feature as the target sound spectrum feature corresponding to the coding segment; the target sound spectrum feature is used to synthesize the speech corresponding to the Chinese text. By adjusting the preset number of decoding times according to the attention weight and the length of the current sound spectrum feature in each round of iterative prediction, it is possible to control the end of the iterative prediction when the number of iterative prediction rounds reaches the preset number of decoding times, so as to obtain accurate target sound spectrum features to improve the accuracy of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1A diagram showing an application environment of a speech synthesis method in an embodiment;

[0041] Figure 2 is a flowchart of a speech synthesis method in one embodiment;

[0042] Figure 3 is a schematic diagram of a sound spectrum prediction model in one embodiment;

[0043] Figure 4 is a structural block diagram of a speech synthesis device in one embodiment;

[0044] Figure 5 is an internal structure diagram of a computer device in one embodiment;

[0045] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] The speech synthesis method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 can determine the coding segment of the sound spectrum to be predicted by encoding the phoneme feature information corresponding to the Chinese text; the server 104 can determine the current sound spectrum feature corresponding to the coding segment predicted in this round in each round of iterative prediction, and the attention weight used when predicting the current sound spectrum feature in this round; the server 104 can adjust the preset number of decoding times corresponding to the coding segment based on the length of the current sound spectrum feature when the attention weight reaches the preset weight threshold, and obtain the adjusted preset number of decoding times; the server 104 can stop the iteration when the number of rounds of iterative prediction reaches the preset number of decodings, and determine the current sound spectrum feature as the target sound spectrum feature corresponding to the coding segment; the target sound spectrum feature is used to synthesize the speech corresponding to the Chinese text. It can be understood that the terminal 102 can display the speech synthesized based on the target feature.

[0048] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0049] In one embodiment, Figure 2 As shown, a speech synthesis method is provided. This embodiment takes the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0050] Step 202, by encoding the phoneme feature information corresponding to the Chinese text, determine the encoding segment of the sound spectrum to be predicted; in each round of iterative prediction, determine the current sound spectrum feature corresponding to the encoding segment predicted in this round, and the attention weight used when predicting the current sound spectrum feature in this round.

[0051] Among them, the phoneme feature information is used to indicate the speech features contained in the Chinese text. It can be understood that the phoneme is the smallest speech unit divided according to the natural attributes of the speech, and is the smallest pronunciation unit in the speech. The Chinese text includes a variety of phoneme features, such as initials, finals, punctuation marks, tones and light tones. These features jointly determine the speech structure corresponding to the Chinese text, that is, how to make the speech corresponding to the Chinese text. The coding segment is part of the coding information corresponding to the Chinese text. It can be understood that encoding the phoneme feature information corresponding to the Chinese text can obtain the coding information corresponding to the Chinese text, and the coding segment is essentially a piece of information in the coding information. The current spectral features include the acoustic features corresponding to the coding segment, such as the frequency information and amplitude information of the sound wave. The attention weight is the intermediate data in the process of predicting the current spectral features corresponding to the coding segment. It can be understood that the attention mechanism is a method for solving problems proposed by imitating human attention. The core goal is to select more critical information for the current task goal from a large amount of information by assigning different weights to some nodes of the neural network. The attention weight can indicate the criticality of the information to extract key features. Text to speech synthesis is the process of converting input Chinese text into speech.

[0052] Specifically, the server can obtain the phoneme feature information corresponding to the Chinese text, and determine the encoding segment of the sound spectrum to be predicted by encoding the phoneme feature information. It can be understood that in each round of iterative prediction, in order to predict the current sound spectrum feature, the intermediate data will be used, and the attention weight is the intermediate data generated in the process of predicting the current sound spectrum feature. In each round of iterative prediction, the server can determine the current sound spectrum feature corresponding to the encoding segment predicted in this round, and the attention weight used when predicting the current sound spectrum feature in this round.

[0053] In one embodiment, the current spectral feature may be at least one of a Mel spectrum and a Mel cepstrum. Mel spectrum (Mel Bank Features), an acoustic feature of a speech signal, contains rich speech information and is often used in speech recognition and speech synthesis.

[0054] Step 204, when the attention weight reaches a preset weight threshold, the preset number of decoding times corresponding to the coding segment is adjusted based on the length of the current spectral feature to obtain an adjusted preset number of decoding times.

[0055] Among them, the preset weight threshold is a threshold set for the attention weight.

[0056] Specifically, the degree of alignment between the current sound spectrum feature and the coding segment is positively correlated with the attention weight. It can be understood that the greater the attention weight, the higher the degree of alignment. When the attention weight reaches the preset weight threshold, it can be determined that the current sound spectrum feature and the coding segment are aligned to a certain extent. At this time, the server can adjust the preset number of decodings corresponding to the coding segment to obtain the adjusted preset number of decodings. Among them, the adjusted preset number of decodings is greater than the previous preset number of decodings.

[0057] In one embodiment, when the attention weight reaches a preset weight threshold and the length of the current spectral feature matches the length of the coding segment, the preset number of decoding times corresponding to the coding segment is adjusted based on the length of the current spectral feature to obtain an adjusted preset number of decoding times.

[0058] In one embodiment, the length matching between the current sound spectrum feature and the coding segment may mean that the time domain length corresponding to the current sound spectrum feature is greater than the time domain length corresponding to the coding segment. It can be understood that the implementation is not limited to the time domain length, and the frequency domain length can also be used to determine the matching between the length of the current sound spectrum feature and the coding segment.

[0059] Step 206: When the number of iterative prediction rounds reaches the preset number of decoding times, the iteration is stopped, and the current sound spectrum feature is determined as the target sound spectrum feature corresponding to the encoding segment.

[0060] Among them, the target spectral features are used to synthesize the speech corresponding to the Chinese text.

[0061] Specifically, although it can be considered that the current spectral feature is aligned with the coding segment to a certain extent, directly outputting the current spectral feature is not accurate enough. The server can continue to predict the current spectral feature corresponding to the coding segment, and when the number of iterative prediction rounds reaches a preset number of decoding times, stop the iteration and determine the current spectral feature as the target spectral feature corresponding to the coding segment.

[0062] In the above-mentioned speech synthesis method, the coding segment of the sound spectrum to be predicted is determined by encoding the phoneme feature information corresponding to the Chinese text; in each round of iterative prediction, the current sound spectrum feature corresponding to the coding segment predicted in this round and the attention weight used in predicting the current sound spectrum feature in this round are determined; when the attention weight reaches the preset weight threshold, the preset number of decoding times corresponding to the coding segment is adjusted based on the length of the current sound spectrum feature to obtain the adjusted preset number of decoding times; when the number of rounds of iterative prediction reaches the preset number of decoding times, the iteration is stopped, and the current sound spectrum feature is determined as the target sound spectrum feature corresponding to the coding segment; the target sound spectrum feature is used to synthesize the speech corresponding to the Chinese text. By adjusting the preset number of decoding times according to the attention weight and the length of the current sound spectrum feature in each round of iterative prediction, it is possible to control the end of the iterative prediction when the number of rounds of iterative prediction reaches the preset number of decoding times, thereby obtaining accurate target sound spectrum features to improve the accuracy of speech synthesis.

[0063] In one embodiment, encoding the phoneme feature information corresponding to the Chinese text to determine the coding segment of the sound spectrum to be predicted includes: determining the phoneme feature information corresponding to the Chinese text; encoding the phoneme feature information to obtain a text coding sequence corresponding to the Chinese text; and determining the coding segment to be predicted for the sound spectrum from the text coding sequence according to a preset time step.

[0064] The text encoding sequence is the intermediate data that conforms to the corresponding format in the process of predicting the spectral features. It is understandable that the server cannot directly predict the spectral features from the phoneme feature information. The spectral features can only be predicted by converting the phoneme feature information into a format multiple times to extract the features. The encoding segment is a partial encoding sequence in the text encoding sequence.

[0065] Specifically, the server can determine the phoneme feature information in the Chinese text. The server can extract the phoneme feature information through an encoder and compress it into a context vector of a fixed length to obtain a text encoding sequence corresponding to the Chinese text. The server can determine the encoding segment to be subjected to spectrogram prediction from the text encoding sequence according to a preset time step. It can be understood that the text encoding sequence includes multiple encoding segments, and the server can perform spectrogram prediction on the multiple encoding segments in chronological order.

[0066] In one embodiment, the server can recognize 237 phonemes, including normal tones 1 to 5 and common punctuation marks, wherein tone 5 represents a light tone.

[0067] In one embodiment, the server may obtain the phoneme feature information corresponding to the Chinese text by querying a standard phoneme dictionary.

[0068] In one embodiment, the encoder includes a word encoding layer, three convolutional layers, and a bidirectional long short-term memory network layer. Among them, word encoding is to convert text into a numerical matrix that can be recognized and processed by a computer through some data transformation or mapping. In the modeling process, the text information is represented by the numerical matrix and participates in training and calculation. The server can convert the phoneme feature information into a text vector through the word encoding layer, and extract the encoding features of the text vector through three convolutional layers and a bidirectional long short-term memory network layer to obtain a text encoding sequence corresponding to the Chinese text.

[0069] In this embodiment, the phoneme feature information corresponding to the Chinese text is determined and encoded to obtain a text encoding sequence corresponding to the Chinese text. Then, according to a preset time step, the encoding segment to be subjected to spectrogram prediction is determined from the text encoding sequence. Subsequently, the spectrogram prediction can be performed on the encoding segment, thereby realizing speech synthesis of the Chinese text.

[0070] In one embodiment, the method further includes: when the number of iterative prediction rounds does not reach the preset number of decoding times, taking the next round as the current round, returning to the step of determining the current spectral features corresponding to the coding segment predicted in the current round, and the attention weight used when predicting the current spectral features in the current round, and continuing to execute.

[0071] Specifically, when the attention weight reaches the preset weight threshold, it is necessary to predict one more part to improve the accuracy of the spectrum prediction. It is understandable that the part predicted more can be a silent feature. When the number of iterative prediction rounds does not reach the preset number of decoding times, the server can use the next round as the current round, return to determine the current spectrum feature corresponding to the coding segment predicted in this round, and continue to execute the step of the attention weight used when predicting the current spectrum feature in this round.

[0072] In one embodiment, when the attention weight reaches a preset weight threshold and the length of the current spectral feature matches the length of the encoding segment, the server can directly predict a part of the silence feature for the current spectral feature to obtain the target spectral feature.

[0073] In one embodiment, the length of the current sound spectrum feature matches the number of rounds of iterative prediction. The server can determine whether the number of rounds of iterative prediction reaches the preset number of decoding times by comparing the length of the current sound spectrum feature and the preset number of decoding times. It can be understood that after each round of iterative prediction, the length of the current sound spectrum feature is fixed, so the server can obtain the number of rounds of iterative prediction by determining the length of the current sound spectrum feature.

[0074] In this embodiment, when the number of iterative prediction rounds does not reach the preset number of decoding times, the next round is taken as the current round, and the step of returning to determine the current spectrum features corresponding to the coding segment predicted in this round, and the attention weight used when predicting the current spectrum features in this round is continued. Through multiple rounds of iterative prediction, the accuracy of spectrum prediction is improved.

[0075] In one embodiment, the method also includes the step of calculating the attention weight; the step of calculating the attention weight includes: in each round of iterative prediction, determining the current attention vector used in predicting the current spectral features in this round; based on the similarity between the current attention vector and the encoding segment, calculating the attention weight corresponding to the current attention vector; the attention weight is used to indicate feature extraction of the attention vector to obtain the current feature vector; the current feature vector is used to predict the current spectral features.

[0076] Specifically, in each round of iterative prediction, the server can determine the current attention vector used in predicting the current spectral features in this round, calculate the similarity between the current attention vector and the encoding segment through the attention recurrent neural network layer, obtain the similarity score, and use the similarity score as the input of the normalized exponential function (softmax function) to obtain the attention weight corresponding to the current attention vector. Among them, the recurrent neural network (RNN) is a type of recurrent neural network that takes sequence data as input, recursively in the direction of sequence evolution, and all nodes (recurrent units) are connected in a chain.

[0077] In one embodiment, the server can combine the two dimensions of context information and position to calculate the similarity score between the current attention vector and the coding segment. It can be understood that the text coding sequence includes complete context information, and the text coding sequence contains corresponding sequence position information. The text content information and state position information contained in the attention vector should be consistent with the context information and sequence position information in the coding segment. The higher the consistency between the two, the higher the similarity score.

[0078] In one embodiment, the server can perform spectrum prediction on the encoded segment through a decoder. The decoder may include a preprocessing layer, an attention recurrent neural network layer, and a decoding recurrent neural network layer. The server can determine the last spectrum feature predicted in the last round, as well as the last attention vector and the last feature vector used in the last round of prediction. The server can input the last spectrum feature into the preprocessing layer, and after splicing the output of the preprocessing layer with the last attention vector, input it into the attention recurrent neural network layer to obtain the current attention vector required for this round of prediction. The server can input the current attention vector and the last feature vector into the decoding recurrent neural network layer to obtain the current feature vector required for this round of prediction.

[0079] In one embodiment, the server may input the current feature vector into the frame prediction network to obtain the current sound spectrum feature. The current sound spectrum feature may include at least one frame of Mel spectrum. It is understood that the server may predict the Mel spectrum of the preset number of spectrum frames in each round of iterative prediction according to the preset number of spectrum frames. For example, the preset number of spectrum frames may be 3 frames, 4 frames or 5 frames.

[0080] In one embodiment, the server may determine the current coding segment to be subjected to spectrum prediction. When performing the first round of spectrum prediction on the current coding segment, the server may determine the last attention vector corresponding to the last target spectrum feature predicted in the last round. The server may input the current coding segment and the last attention vector into the attention recurrent neural network layer to obtain the current attention vector required for the current round of prediction.

[0081] In this embodiment, in each round of iterative prediction, the current attention vector used in predicting the current spectrogram features in this round is determined; based on the similarity between the current attention vector and the encoding segment, the attention weight corresponding to the current attention vector is calculated; and subsequently, the preset number of decoding times can be adjusted based on the attention weight, so as to more accurately predict the spectrogram and improve the accuracy of speech synthesis.

[0082] In one embodiment, when the attention weight reaches a preset weight threshold, the preset number of decoding times corresponding to the coding segment is adjusted based on the length of the current spectral feature, and the adjusted preset number of decoding times includes: when the attention weight reaches the preset weight threshold, determining the decoding round offset value corresponding to the coding segment; adjusting the preset number of decoding times corresponding to the coding segment according to the decoding round offset value and the length of the current spectral feature, and obtaining the adjusted preset number of decoding times.

[0083] The decoding round offset value is used to indicate the offset of the preset decoding times.

[0084] Specifically, when the attention weight reaches a preset weight threshold, the server can determine a decoding round offset value corresponding to the coding segment, and add the decoding round offset value to the preset decoding times to obtain an adjusted preset decoding times.

[0085] In one embodiment, the preset number of decoding times before adjustment corresponding to all encoding segments may be a preset fixed value. The server may determine the length of the current sound spectrum feature, and calculate the sum of the decoding round offset value and the length of the current sound spectrum feature to obtain the adjusted preset number of decoding times.

[0086] In this embodiment, when the attention weight reaches a preset weight threshold, the decoding round offset value corresponding to the coding segment is determined; based on the decoding round offset value, the preset number of decoding times corresponding to the coding segment is adjusted to obtain the adjusted preset number of decoding times, thereby controlling the iteration stop based on the preset number of decoding times to perform spectrogram prediction more accurately.

[0087] In one embodiment, the method further includes: predicting the end identification information corresponding to the coding segment based on the current spectral features; when the end identification information cannot meet the preset end condition, comparing the number of iterative prediction rounds with the preset number of decoding times, and executing the step of stopping the iteration when the number of iterative prediction rounds reaches the preset number of decoding times.

[0088] The end identification information is used to indicate the probability of the end of the spectrum prediction.

[0089] Specifically, the server can determine the current feature vector corresponding to the current spectral feature, and input the current feature vector into the end identification information corresponding to the linear projection layer prediction coding segment. The server can determine the length of the current spectral feature when the end probability indicated by the end identification information cannot reach the preset end threshold. It can be understood that the length of the current spectral feature can indicate the number of iterative prediction rounds in which the current spectral feature is located. The server can determine the number of rounds of iterative prediction up to the current time based on the length of the current spectral feature, compare the number of iterative prediction rounds with the adjusted preset number of decoding times, and execute the step of stopping the iteration when the number of iterative prediction rounds reaches the preset number of decoding times.

[0090] In one embodiment, the length value of the current sound spectrum feature may be consistent with the round value of the iterative prediction. The server may directly compare the length of the current sound spectrum feature with the adjusted preset number of decoding times. When the length of the current sound spectrum feature matches the adjusted preset number of decoding times, the server may execute the step of stopping iteration when the number of iterative prediction rounds reaches the preset number of decoding times. It can be understood that the matching situation may be the situation where the length value of the current sound spectrum feature is consistent with the preset number of decoding times.

[0091] In one embodiment, when the end identification information cannot meet the preset end condition, the server can return to the step of adjusting the preset number of decoding times corresponding to the encoding segment based on the length of the current spectral feature when the attention weight reaches the preset weight threshold, and obtain the adjusted preset number of decoding times to continue iterating.

[0092] In one embodiment, the server may input the current feature vector into the frame prediction network and the linear projection layer respectively to obtain the end identification information corresponding to the current spectrum feature and the prediction coding segment.

[0093] In one embodiment, the server can determine whether the attention weight reaches a preset weight threshold and whether the length of the current spectral feature matches the length of the encoding segment when the end identification information fails to meet the preset end condition.

[0094] In this embodiment, when the end identification information cannot meet the preset end condition, the number of iterative prediction rounds is compared with the preset number of decoding times, and the step of stopping the iteration is executed when the number of iterative prediction rounds reaches the preset number of decoding times. The iteration stop is controlled by the standards of the two dimensions of the end identification information and the preset number of decoding times, thereby improving the accuracy of the spectrum prediction.

[0095] In one embodiment, the method further includes: when the end identification information satisfies a preset end condition, stopping iteration, and determining the current sound spectrum feature as the target sound spectrum feature corresponding to the encoding segment.

[0096] Specifically, when the end identification information satisfies a preset end condition and the length of the current spectrum feature matches the length of the current encoding segment, the server can control the iterative prediction to stop and determine the current spectrum feature as the target spectrum feature corresponding to the encoding segment.

[0097] In one embodiment, when the end identification information satisfies a preset end condition and the length of the current sound spectrum feature matches the length of the current encoding segment, the server may stop iteration and determine the current sound spectrum feature as the target sound spectrum feature corresponding to the encoding segment.

[0098] In this embodiment, when the end identification information meets the preset end condition, the iteration is stopped, and the current sound spectrum feature is determined as the target sound spectrum feature corresponding to the encoding segment, which can avoid the problem of the sound spectrum length being too short when the preset end condition is met.

[0099] In one embodiment, Figure 3The figure shows a schematic diagram of the spectrogram prediction model. The server can perform spectrogram prediction on Chinese text through the spectrogram prediction model. The spectrogram prediction model is a sequence-to-sequence generation model with a local sensitive attention mechanism. It can be understood that in the task of Chinese text-to-speech synthesis, the local sensitive attention mechanism reasonably distributes the weight of each element in the sequence, so that the model pays different attention to each part of the sequence and pays more attention to the part that is more closely related to the synthesized content. The spectrogram prediction model includes an encoder, a decoder, and a post-processing network. The encoder includes a word encoding layer, three convolutional layers, and a bidirectional long short-term memory network layer. The decoder includes a preprocessing layer, an attention recurrent neural network layer, and a decoding recurrent neural network layer. The post-processing network is used to improve the result of spectrum reconstruction. The result of spectrum reconstruction is the target spectrogram feature output by the decoder.

[0100] The server can input the current feature vector output by the decoding recurrent neural network layer into the frame prediction network and the linear projection layer respectively to obtain three frames of Mel spectrum and corresponding end identification information. The server can input at least one frame of the three frames of Mel spectrum into the preprocessing layer for the next round of iteration until the iteration ends. The server can input the final predicted Mel spectrum into the post-processing network, predict a residual term, and superimpose the residual term into the Mel spectrum to obtain an improved Mel spectrum.

[0101] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0102] Based on the same inventive concept, the embodiment of the present application also provides a speech synthesis device for implementing the speech synthesis method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more speech synthesis device embodiments provided below can refer to the limitations on the speech synthesis method above, and will not be repeated here.

[0103] In one embodiment, Figure 4 As shown, a speech synthesis device 400 is provided, including: an encoding module 402 and a decoding module 404, wherein:

[0104] The encoding module 402 is used to determine the encoding segment of the sound spectrum to be predicted by encoding the phoneme feature information corresponding to the Chinese text;

[0105] The decoding module 404 determines, in each round of iterative prediction, the current spectral feature corresponding to the coding segment predicted in this round, and the attention weight used when predicting the current spectral feature in this round; when the attention weight reaches a preset weight threshold, the preset number of decoding times corresponding to the coding segment is adjusted based on the length of the current spectral feature to obtain the adjusted preset number of decoding times; when the number of iterative prediction rounds reaches the preset number of decoding times, the iteration is stopped, and the current spectral feature is determined as the target spectral feature corresponding to the coding segment; the target spectral feature is used to synthesize the speech corresponding to the Chinese text.

[0106] In one embodiment, the encoding module 402 is further used to determine the phoneme feature information corresponding to the Chinese text; encode the phoneme feature information to obtain a text encoding sequence corresponding to the Chinese text; and determine the encoding segment to be subjected to spectrogram prediction from the text encoding sequence according to a preset time step.

[0107] In one embodiment, the decoding module 404 is also used to, when the number of iterative prediction rounds does not reach a preset number of decoding times, take the next round as the current round, return to the step of determining the current spectral features corresponding to the coding segment predicted in the current round, and the attention weight used when predicting the current spectral features in the current round, and continue to execute.

[0108] In one embodiment, the decoding module 404 is also used to determine the current attention vector used in predicting the current spectral features in each round of iterative prediction; based on the similarity between the current attention vector and the encoding segment, the attention weight corresponding to the current attention vector is calculated; the attention weight is used to indicate feature extraction of the attention vector to obtain the current feature vector; the current feature vector is used to predict the current spectral features.

[0109] In one embodiment, the decoding module 404 is further used to determine a decoding round offset value corresponding to the coding segment when the attention weight reaches a preset weight threshold; and adjust the preset number of decoding times corresponding to the coding segment according to the decoding round offset value and the length of the current spectral feature to obtain an adjusted preset number of decoding times.

[0110] In one embodiment, the decoding module 404 is also used to predict the end identification information corresponding to the coding segment based on the current sound spectrum features; when the end identification information cannot meet the preset end condition, the number of iterative prediction rounds is compared with the preset number of decoding times, and the step of stopping the iteration is executed when the number of iterative prediction rounds matches the preset number of decoding times.

[0111] In one embodiment, the decoding module 404 is further used to stop the iteration when the end identification information meets the preset end condition, and determine the current sound spectrum feature as the target sound spectrum feature corresponding to the encoding segment. Each module in the above-mentioned speech synthesis device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0112] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the spectral features predicted by each round of iteration. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech synthesis method is implemented.

[0113] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a speech synthesis method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0114] Those skilled in the art will understand that Figure 5 and Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0115] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0116] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0117] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0118] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0119] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0120] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0121] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A speech synthesis method, characterized in that: The method comprises: By encoding the phoneme feature information corresponding to the Chinese text, the encoding segment of the sound spectrum to be predicted is determined; In each round of iterative prediction, determining the current spectral feature corresponding to the encoding segment predicted in this round, and the attention weight used when predicting the current spectral feature in this round; When the attention weight reaches a preset weight threshold, determining a decoding round offset value corresponding to the coding segment; adjusting a preset number of decoding times corresponding to the coding segment according to the decoding round offset value and the length of the current spectrum feature to obtain an adjusted preset number of decoding times; When the number of iterative prediction rounds reaches the preset number of decoding times, the iteration is stopped, and the current sound spectrum feature is determined as the target sound spectrum feature corresponding to the encoding segment; the target sound spectrum feature is used to synthesize the speech corresponding to the Chinese text; Wherein, in each round of iterative prediction, determining the current spectral features corresponding to the coding segment predicted in this round, and the attention weight used when predicting the current spectral features in this round, includes: In each round of iterative prediction, determine the current attention vector used when predicting the current spectral features in this round; Based on the similarity between the current attention vector and the encoding segment, an attention weight corresponding to the current attention vector is calculated; the attention weight is used to indicate feature extraction of the attention vector to obtain a current feature vector; the current feature vector is used to predict current spectrum features.

2. The method according to claim 1, characterized in that The step of encoding the phoneme feature information corresponding to the Chinese text to determine the encoding segment of the sound spectrum to be predicted comprises: Determine phoneme feature information corresponding to the Chinese text; Encoding the phoneme feature information to obtain a text encoding sequence corresponding to the Chinese text; According to the preset time step, the coding segment to be subjected to the spectrogram prediction is determined from the text coding sequence.

3. The method according to claim 2, characterized in that The step of encoding the phoneme feature information to obtain a text encoding sequence corresponding to the Chinese text includes: The phoneme feature information is converted into a text vector through a word encoding layer, and encoding features of the text vector are extracted through three convolutional layers and a bidirectional long short-term memory network layer to obtain a text encoding sequence corresponding to the Chinese text; The word encoding layer is used to convert text into a numerical matrix that can be recognized and processed by a computer through data transformation or mapping, and the text information represented by the numerical matrix participates in training and calculation.

4. The method according to claim 1, characterized in that: The method further comprises: When the number of iterative prediction rounds does not reach the preset number of decoding times, the next round is taken as the current round, and the step of returning to determine the current spectral features corresponding to the coding segment predicted in the current round, and the attention weight used when predicting the current spectral features in the current round is continued.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Predicting end identification information corresponding to the encoding segment based on the current sound spectrum feature; When the end identification information cannot meet the preset end condition, the number of rounds of iterative prediction is compared with the preset number of decoding times, and the step of stopping iteration is performed when the number of rounds of iterative prediction reaches the preset number of decoding times.

6. The method according to claim 5, characterized in that The method further comprises: When the end identification information satisfies a preset end condition, the iteration is stopped, and the current sound spectrum feature is determined as the target sound spectrum feature corresponding to the encoding segment.

7. A speech synthesis device, characterized in that: The device comprises: An encoding module, used to determine the encoding segment of the sound spectrum to be predicted by encoding the phoneme feature information corresponding to the Chinese text; A decoding module, used to determine, in each round of iterative prediction, the current spectral feature corresponding to the encoding segment predicted in this round, and the attention weight used when predicting the current spectral feature in this round; The decoding module is used to determine the decoding round offset value corresponding to the coding segment when the attention weight reaches the preset weight threshold; adjust the preset number of decoding times corresponding to the coding segment according to the decoding round offset value and the length of the current sound spectrum feature; stop iteration when the number of iterative prediction rounds reaches the preset number of decoding times, and determine the current sound spectrum feature as the target sound spectrum feature corresponding to the coding segment; the target sound spectrum feature is used to synthesize the speech corresponding to the Chinese text; In terms of determining the current spectral features corresponding to the coding segment predicted in each round of iterative prediction, and the attention weight used in predicting the current spectral features in this round, the decoding module is used to determine the current attention vector used in predicting the current spectral features in this round in each round of iterative prediction; calculate the attention weight corresponding to the current attention vector based on the similarity between the current attention vector and the coding segment; the attention weight is used to indicate feature extraction of the attention vector to obtain the current feature vector; and the current feature vector is used to predict the current spectral features.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images