Speech synthesis method and apparatus, storage medium, electronic device

Abstract

The present disclosure provides a speech synthesis method and device, a storage medium and an electronic device, and relates to the technical field of information processing. The method comprises the following steps: obtaining a symbol sequence of a to-be-synthesized sentence; using a pre-trained acoustic prediction model to perform acoustic feature prediction on the symbol sequence to obtain acoustic features corresponding to the to-be-synthesized sentence; the acoustic prediction model comprises a prosody prediction model, and the prosody prediction model learns prosodic features of a reference recording audio in a model training stage to enhance prosodic features of the to-be-synthesized sentence in a speech synthesis stage; and performing feature conversion and synthesis on the acoustic features to obtain speech corresponding to the to-be-synthesized sentence. The present disclosure can solve the problems that a speech synthesis system in the related art cannot meet the needs of prosodic naturalness and expressiveness of specific business scenarios and the speech synthesis effect is poor.

Description

Technical Field

[0001] This disclosure relates to the field of information processing technology, and more specifically, to a speech synthesis method and apparatus, a storage medium, and an electronic device. Background Technology

[0002] Speech synthesis technology has been widely used in various human-computer interaction and intelligent voice devices. Speech synthesis systems can realize the function of text-to-speech (TTS). Online TTS services basically meet the stability level of synthesized audio in terms of rhythmic pauses, timbre similarity, pronunciation accuracy, and sound quality.

[0003] However, with the diversification of business scenarios, in some specific scenarios (such as those requiring conversational style, strong rhythm, or live streaming), the synthesis results of existing TTS systems tend to be averaged, which cannot meet the requirements of these business scenarios for naturalness of rhythm and expressiveness, resulting in poor speech synthesis quality.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a speech synthesis method and apparatus, storage medium, and electronic device, thereby solving to some extent the problems in the related art where TTS systems cannot meet the requirements of naturalness of rhythm and expressiveness in specific business scenarios and the speech synthesis effect is poor.

[0006] According to a first aspect of this disclosure, a speech synthesis method is provided, the method comprising: acquiring a symbol sequence of a statement to be synthesized, the statement to be synthesized including text to be synthesized and a query result statement for a target object; using a pre-trained acoustic prediction model to predict acoustic features of the symbol sequence to obtain acoustic features corresponding to the statement to be synthesized; the acoustic prediction model including a prosodic prediction model, the prosodic prediction model learning prosodic features of a reference audio recording during the model training phase to enhance the prosodic features of the statement to be synthesized during the speech synthesis phase; and performing feature transformation and synthesis on the acoustic features to obtain speech corresponding to the statement to be synthesized.

[0007] Optionally, the acoustic prediction model further includes an encoding model and a decoding model. The step of using the pre-trained acoustic prediction model to predict the acoustic features of the symbol sequence includes: using the pre-trained encoding model to encode the symbol sequence once to obtain a first encoding vector; using the pre-trained prosody prediction model to predict the prosody features of the first encoding vector to obtain a prosody feature vector; and predicting the acoustic features of the sentence to be synthesized based on the pre-trained decoding model, the first encoding vector, and the prosody feature vector to obtain the acoustic features corresponding to the sentence to be synthesized.

[0008] Optionally, predicting the acoustic features of the sentence to be synthesized based on the pre-trained decoding model, the first encoding vector, and the prosodic feature vector includes: using a pre-trained variable adaptation model to perform variable prediction on the superposition result of the first encoding vector and the prosodic feature vector to obtain a first variable prediction result; and using a pre-trained decoding model to perform attention-based decoding processing on the first variable prediction result to obtain the acoustic features corresponding to the sentence to be synthesized.

[0009] Optionally, predicting the acoustic features of the sentence to be synthesized based on the pre-trained decoding model, the first encoding vector, and the prosodic feature vector includes: inputting the first encoding vector and the prosodic feature vector into a pre-trained variable adaptation model to perform variable prediction and obtain a second variable prediction result; and using the pre-trained decoding model to perform attention-based decoding processing on the second variable prediction result to obtain the acoustic features corresponding to the sentence to be synthesized.

[0010] Optionally, the prosody prediction model includes a first prosody prediction model and a second prosody prediction model. The step of predicting prosody features on the first encoding vector using the pre-trained prosody prediction model includes: using the first prosody prediction model to predict sentence-level prosody features on the first encoding vector to obtain a first prosody feature vector; using the second prosody prediction model to predict phoneme-level prosody features on the superimposed features of the first encoding vector and the first prosody feature vector to obtain a second prosody feature vector; the step of predicting variables on the superimposed result of the first encoding vector and the prosody feature vector includes: predicting variables on the superimposed result of the second prosody feature vector and the superimposed features.

[0011] Optionally, the prosody prediction model includes a first prosody prediction model and a second prosody prediction model. The step of using the pre-trained prosody prediction model to predict prosody features on the first encoding vector includes: using the first prosody prediction model to predict sentence-level prosody features on the first encoding vector to obtain a third prosody feature vector; and using the second prosody prediction model to predict phoneme-level prosody features on the first encoding vector to obtain a fourth prosody feature vector. The step of inputting the first encoding vector and the prosody feature vector into a pre-trained variable adaptation model for variable prediction includes: inputting the first encoding vector, the third prosody feature vector, and the fourth prosody feature vector into the pre-trained variable adaptation model for variable prediction.

[0012] Optionally, the sentence-level prosodic feature prediction includes: performing temporal feature processing and linear transformation on the input data of the first prosodic prediction model.

[0013] Optionally, the phoneme-level prosodic feature prediction includes: performing convolution processing and linear transformation on the input data of the second prosodic prediction model.

[0014] Optionally, the statement to be synthesized may also include the recorded statement of the target object, and the method may further include: using a pre-trained first reference coding model to perform secondary coding processing on the recorded statement to obtain a second coding vector.

[0015] The variable prediction includes: using a pre-trained variable adaptation model to predict variables by combining the superposition result of the second prosodic feature vector and the superimposed feature with the second encoding vector; or using a pre-trained variable adaptation model to predict variables by combining the first encoding vector, the prosodic feature vector and the second encoding vector.

[0016] Optionally, the method further includes training the acoustic prediction model. The training process includes: acquiring training samples, which include recording samples and corresponding acoustic feature samples and symbol sequence samples; using the training samples to train the initial acoustic prediction model once to obtain an intermediate model; the initial acoustic prediction model includes a first reference coding model and a second reference coding model; fixing the model parameters of the first reference coding model and the second reference coding model of the intermediate model, and using the training samples, the first reference coding model, and the second reference coding model to perform a second training on the acoustic prediction model.

[0017] Optionally, the initial acoustic prediction model further includes an encoding model and a decoding model. Training the initial acoustic prediction model once using the training samples includes: encoding symbol sequence samples using the encoding model to obtain a first sample encoding vector; encoding recording samples using the first reference encoding model to obtain a second sample encoding vector; encoding acoustic feature samples using the second reference encoding model to obtain a third sample encoding vector; performing feature superposition on the first and second sample encoding vectors, and then concatenating them with the third sample encoding vector to obtain concatenated features; decoding the concatenated features using the decoding model and outputting the result; calculating a first loss function value based on the output result and the acoustic feature samples; and adjusting the model parameters of the encoding model, the first reference encoding model, the second reference encoding model, and the decoding model based on the first loss function value.

[0018] Optionally, the acoustic prediction model includes a first prosody prediction model and a second prosody prediction model. The secondary training of the acoustic prediction model includes: encoding symbol sequence samples using a first-trained encoding model to obtain a fourth sample encoding vector; encoding recording samples using a first-trained first reference encoding model to obtain a fifth sample encoding vector; encoding acoustic feature samples using a second-trained second reference encoding model to obtain a sixth sample encoding vector; superimposing the fourth and fifth sample encoding vectors and inputting them into the first and second prosody prediction models respectively; calculating a second loss function value based on the fifth sample encoding vector and the output of the first prosody prediction model; adjusting the model parameters of the first prosody prediction model based on the second loss function value; calculating a third loss function value based on the sixth sample encoding vector and the output of the second prosody prediction model; and adjusting the model parameters of the second prosody prediction model based on the third loss function value.

[0019] According to a second aspect of this disclosure, a speech synthesis apparatus is provided, the apparatus comprising: an acquisition module, a prediction module, and a speech synthesis module; the acquisition module is configured to acquire a symbol sequence of a statement to be synthesized, the statement to be synthesized including text to be synthesized and a query result statement for a target object; the prediction module is configured to use a pre-trained acoustic prediction model to predict acoustic features of the symbol sequence to obtain acoustic features corresponding to the statement to be synthesized; the acoustic prediction model includes a prosodic prediction model, which learns prosodic features of a reference audio recording during the model training phase to enhance the prosodic features of the statement to be synthesized during the speech synthesis phase; the speech synthesis module is configured to perform feature conversion and synthesis on the acoustic features to obtain speech corresponding to the statement to be synthesized.

[0020] According to a third aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0021] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: one or more processors; and a storage device for one or more programs, which, when executed by the one or more processors, cause the one or more processors to perform the method described in any of the above embodiments.

[0022] The exemplary embodiments disclosed herein may have some or all of the following beneficial effects:

[0023] In the speech synthesis method provided by the exemplary embodiments of this disclosure, on the one hand, a prosodic prediction model can be added to the acoustic prediction model. By learning the prosodic features of the reference recorded audio during the model training stage, the prosodic features of the sentence to be synthesized can be enhanced during the speech synthesis stage. This results in prosodic-enhanced speech after acoustic feature transformation and synthesis, meeting the requirements for prosodic expression in specific business scenarios (scenarios with high prosodic expression requirements) and improving the accuracy and realism of the synthesized speech. On the other hand, by using the text to be synthesized and the query result statement targeting the target object, it is ensured that the pre-trained acoustic prediction model can synthesize the speech of the sentence to be synthesized for the specific target object, realizing speech customization for the target object in specific business scenarios.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0026] Figure 1 The illustration shows an exemplary application scenario architecture diagram of a speech synthesis method and apparatus according to an embodiment of the present disclosure.

[0027] Figure 2 One of the flowcharts of a speech synthesis method according to an embodiment of the present disclosure is illustrated schematically.

[0028] Figure 3 One of the flowcharts illustrating an acoustic feature prediction process according to one embodiment of the present disclosure is shown schematically.

[0029] Figure 4 A second flowchart illustrating an acoustic feature prediction process according to one embodiment of the present disclosure is shown schematically.

[0030] Figure 5 One of the schematic diagrams illustrating variable prediction of a variable fitting model according to one embodiment of the present disclosure is shown.

[0031] Figure 6 One of the schematic diagrams illustrates the variable prediction of a pitch predictor in a variable adaptation model according to an embodiment of the present disclosure.

[0032] Figure 7 A flowchart of the acoustic feature prediction process according to one embodiment of the present disclosure is illustrated in Figure 3.

[0033] Figure 8 A flowchart of an acoustic feature prediction process according to one embodiment of the present disclosure is illustrated in diagram four.

[0034] Figure 9 The illustration shows a second schematic diagram of variable prediction using a variable fitting model according to one embodiment of the present disclosure.

[0035] Figure 10 The diagram illustrates a second schematic of variable prediction by a pitch predictor in a variable adaptation model according to an embodiment of the present disclosure.

[0036] Figure 11 The flowchart illustrating the training process of an acoustic prediction model according to an embodiment of the present disclosure is shown.

[0037] Figure 12 The diagram schematically illustrates a flowchart of the first part of the training process of an acoustic prediction model according to an embodiment of the present disclosure.

[0038] Figure 13 The flowchart illustrating the second part of the training process of an acoustic prediction model according to an embodiment of the present disclosure is shown schematically.

[0039] Figure 14 A schematic block diagram of a speech synthesis apparatus according to one embodiment of the present disclosure is shown.

[0040] Figure 15 A block diagram of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0041] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0042] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0043] like Figure 1 As shown in Figure 100, an exemplary system scenario for the application of a speech synthesis method and apparatus is provided. System 100 may include a terminal 110 and a server 120. This embodiment illustrates the application of the method to server 120. It is understood that the method can also be applied to a terminal, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. Server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. It can also be a node in a blockchain. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, in-vehicle device, etc., but is not limited to these. For example, during human-computer interaction, the terminal can be a smart device, and the user performs speech synthesis through the smart device. When the speech synthesis method provided in this embodiment is implemented through interaction between the terminal and the server, the terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0044] In this example, the user can input the text to be synthesized on terminal 110, or the user can perform specific operations on the terminal to generate the text (such as clicking the "Read" button or the "Recite" button on the client page). Terminal 110 sends the text to be synthesized to server 120, so that the server can obtain the corresponding speaker's query result statement based on the text to be synthesized, and perform symbol serialization on the text to be synthesized and the query result statement to form a symbol sequence. Using a pre-trained acoustic prediction model, acoustic features are predicted on the symbol sequence to obtain the acoustic features corresponding to the statement to be synthesized. The acoustic prediction model includes a prosodic prediction model, which learns the prosodic features of the reference audio recording during the model training phase to enhance the prosodic features of the statement to be synthesized during the speech synthesis phase. Feature transformation and synthesis are performed on the acoustic features to obtain the speech corresponding to the statement to be synthesized. The synthesized speech is then sent to the terminal for playback.

[0045] The speech synthesis method provided in this disclosure can be executed on server 120, and correspondingly, the speech synthesis device is generally installed in server 120. The speech synthesis method provided in this disclosure can also be executed on terminal 110, and correspondingly, the speech synthesis device is generally installed in terminal 110.

[0046] The speech synthesis method disclosed in the embodiments of this specification will now be described in conjunction with specific examples.

[0047] refer to Figure 2 As shown, a speech synthesis method according to an example embodiment of the present disclosure may include the following steps S210-S230.

[0048] Step S210: Obtain the symbol sequence of the statement to be synthesized.

[0049] In this example implementation, the symbol sequence may include a factor sequence or a sequence of pinyin letters. The sentence to be synthesized may include the text to be synthesized and a query result sentence targeting a specific object. The target object may be a specified speaker, and the speaker's sentences can be searched from a corpus based on the determined speaker. For example, identification information can be added to each speaker, and the corresponding sentences can be retrieved through the identification information. The text to be synthesized or the query result sentence may be converted into pinyin, and then the pinyin may be converted into phonemes, forming a symbol sequence according to the text order.

[0050] In this example implementation, the text to be synthesized can be user input, generated based on user operations, or obtained after processing the user input, such as searching for the text to be synthesized from the user's voice input. This example does not limit this.

[0051] Step S220: Using a pre-trained acoustic prediction model, acoustic features are predicted on the symbol sequence to obtain the acoustic features corresponding to the sentence to be synthesized. The acoustic prediction model includes a prosody prediction model, which learns the prosody features of the reference audio recording during the model training stage to enhance the prosody features of the sentence to be synthesized during the speech synthesis stage.

[0052] In this example implementation, the acoustic prediction model may include a transformer-based acoustic model and a prosodic prediction model. The transformer-based acoustic model can predict the basic prosody, prosodic pauses, timbre, pronunciation, and timbre of the synthesized utterance. The backbone network of the acoustic prediction model is an end-to-end network architecture based on an encoder-decoder structure, such as Fastspeech2. The prosodic prediction model adds prosodic enhancement features to the encoded features of this backbone network to enhance prosodic expressiveness.

[0053] In this example implementation, the acoustic features can be spectrograms, such as Mel-Spectrogram, MFCC (Mel-Frequency Cepstral Coefficient), linear spectrograms, such as STFT (Short-Time Fourier Transform) spectrograms, etc., and this example does not limit them.

[0054] In this example implementation, the reference audio recording can be a sentence with strong pauses and a distinct prosodic style. This audio recording can originate from different speakers, and can be recordings of speakers of different genders and ages; this example does not impose any limitations on this. By learning various prosodic styles during the training phase, the prosodic prediction model can better predict the prosodic style of the sentence to be synthesized during speech synthesis.

[0055] Step S230: Perform feature conversion and synthesis on the acoustic features to obtain the speech corresponding to the sentence to be synthesized.

[0056] In this example implementation, a vocoder can be used to acoustically encode the acoustic spectrum to synthesize the speech corresponding to the sentence to be synthesized, and the speech has prosodic features.

[0057] In the speech synthesis method provided in this exemplary embodiment, on the one hand, a prosodic prediction model can be added to the acoustic prediction model. By learning the prosodic features of the reference audio recording during the model training phase, the prosodic features of the sentence to be synthesized can be enhanced during the speech synthesis phase. This results in prosodic-enhanced speech after acoustic feature transformation and synthesis, meeting the requirements for prosodic expression in specific business scenarios (scenarios with high prosodic performance requirements) and improving the accuracy and realism of the synthesized speech. On the other hand, by using the text to be synthesized and the query result statement targeting the target object, it is ensured that the pre-trained acoustic prediction model can synthesize the speech of the sentence to be synthesized for the specific target object, realizing speech customization for the target object in specific business scenarios.

[0058] The steps of this disclosure are described in more detail below.

[0059] In some embodiments, reference Figure 3 The acoustic prediction model may include an encoding model 310, a prosody prediction model 320, and a decoding model 330. Using a pre-trained acoustic prediction model, acoustic feature prediction of a symbol sequence may include the following steps.

[0060] The first step is to use a pre-trained encoding model to encode the symbol sequence once to obtain the first encoding vector.

[0061] In this example implementation, the encoding model 310 may include a phoneme embedding module, a multi-head attention encoder, and a speaker embedding module. The text to be synthesized can be converted into phoneme symbols, forming a first symbol sequence in text order. The first symbol sequence is input into the phoneme embedding module for dimension mapping to obtain a first embedding vector. The first embedding vector undergoes encoding processing based on a multi-head attention mechanism through the multi-head attention encoder to extract contextual semantic features. In this example, the first embedding vector can also be positionally encoded before being input into the multi-head attention encoder to record the position of each symbol in the sequence.

[0062] In this example implementation, the text corresponding to the query result statement for the target object is converted into phonetic symbols to form a corresponding second symbol sequence. This second symbol sequence is then input into the speaker embedding module for dimension mapping to obtain a second embedding vector.

[0063] In this example implementation, a single encoding process may include embedding of the text to be synthesized, encoding based on a multi-head attention mechanism, and embedding of the query result statement of the target object. It may also include processes such as feature overlay and positional encoding, but this example does not limit these processes.

[0064] The second step involves using a pre-trained prosody prediction model to predict prosody features from the first encoded vector, thereby obtaining the prosody feature vector.

[0065] In this example implementation, the prosody prediction model 320 may include a first prosody prediction model and a second prosody prediction model. The first prosody prediction model may be an utterance-level prosody predictor, and the second prosody prediction model may be a phoneme-level prosody predictor.

[0066] For example, a first prosodic prediction model can be used to predict the prosodic features of the first encoding vector at the sentence level to obtain a first prosodic feature vector; then, a second prosodic prediction model can be used to predict the phoneme-level prosodic features of the superimposed features of the first encoding vector and the first prosodic feature vector to obtain a second prosodic feature vector.

[0067] In this example implementation, the output of the encoding model can be input into the first prosodic prediction model to enhance the prosodicity of each sentence. Then, the outputs of the first and encoding models are superimposed and input into the second prosodic prediction model to enhance the prosodicity of each phoneme within the sentence. Finally, the output of the second prosodic prediction model is concatenated with its input data to merge various features into the backbone network. Alternatively, the concatenated features can be input into a fully connected layer for feature integration.

[0068] In this example implementation, the first prosody prediction model may include a recurrent neural network (such as a gated recurrent neural network, GRU) and linear layers. The second prosody prediction model may include a one-dimensional convolution-based neural network. Exemplarily, the second prosody prediction model may include multiple convolutional units and linear units (linear layers), each convolutional unit including a one-dimensional convolutional layer and a normalization layer. A linear activation function (ReLU) may be added after the one-dimensional convolutional layer, and dropout may be added after the normalization layer.

[0069] The third step is to predict the acoustic features of the sentence to be synthesized based on the pre-trained decoding model, the first encoding vector, and the prosodic feature vector, thereby obtaining the acoustic features corresponding to the sentence to be synthesized.

[0070] In this example implementation, the decoding model 330 can be a multi-head attention-based decoding model, corresponding to the multi-head attention encoder in the encoding model. Features can be positionally encoded before entering the decoding model to mark sequence positions.

[0071] In this example implementation, before the prosodic-enhanced features (concatenated features) enter the decoding model, variable prediction is performed on them to optimize the prosodic style and timbre of the prosodic-enhanced features. In this example, acoustic variables may include phoneme duration, pitch, amplitude energy, etc., and may also include other acoustic variables, which are not limited in this example. Variable prediction is performed separately for each type of acoustic variable.

[0072] For example, a pre-trained variable adaptation model can be used to predict variables from the superposition of the first encoding vector and the prosodic feature vector to obtain the first variable prediction result. Then, a pre-trained decoding model can be used to decode the first variable prediction result based on an attention mechanism to obtain the acoustic features corresponding to the sentence to be synthesized.

[0073] In this example implementation, the variable adaptation model may include a phoneme duration predictor, a pitch predictor, an amplitude energy predictor, etc. Duration is the length of each phoneme in the logarithmic domain, which can improve segmentation accuracy and reduce the information gap between input and output. For the pitch predictor, the output sequence is a frame-level F0 sequence; for the energy predictor, the output is the energy sequence of each Mel spectrum frame. All predictors share the same model structure but have different model parameters. The predictor corresponding to each acoustic variable may include a fully connected layer, multiple convolutional units, and linear units (linear layers). Each convolutional unit may include a one-dimensional convolutional layer and a normalization layer. A linear activation function (ReLU) can be added after the one-dimensional convolutional layer, and dropout can be added after the normalization layer. For example, the predictor corresponding to each acoustic variable may consist of a fully connected layer, two convolutional units (one-dimensional convolutional layer + ReLU activation function), and a linear layer connected in sequence.

[0074] In this example implementation, each variable predictor in the variable-fitting model has a similar model structure. The hidden sequence can be taken as input and the variation for each phoneme (duration) or each frame (pitch and energy) can be predicted using mean squared error loss.

[0075] In this example implementation, a length regularization unit may also be included to match the length of the input factor sequence with the length of the output Mel spectrum sequence, preparing for the output; simultaneously, the length regularization unit can also control the pronunciation speed of the acoustic prediction model, etc. The variable adaptation model may also include superposition processing, etc., but this example does not limit this.

[0076] In the above embodiments (such as) Figure 3As shown, this disclosure adds a prosodic feature prediction model (such as a first prosodic feature prediction model and a second prosodic feature prediction model) to the backbone network of the acoustic prediction model, and superimposes the prosodic features predicted by the prosodic feature prediction model into the features of the backbone network to enhance the prosodicity at the sentence level and phoneme level, improve the sense of pause, and make the tone of the synthesized audio richer.

[0077] Considering that the added prosodic reference information may have an adverse effect on the timbre of the backbone network, that is, the original timbre may be changed while prosodic is added, this disclosure solves the problem through the following solution.

[0078] In some embodiments, such as Figure 4 As shown, the prosody prediction model is decoupled from the backbone network. The output of the prosody prediction model is directly input into the variable adaptation model, without superimposing the prediction results of the prosody prediction model with the features of the backbone network, thus avoiding the influence of the prosody prediction model on the timbre synthesis effect of the backbone network. Because this scheme... Figure 3 Since the backbone network of the scheme is the same, this example will explain in detail the different ways of adding the prosody prediction model. The prosody prediction model includes a first prosody prediction model and a second prosody prediction model. Using the pre-trained prosody prediction model, the prosody feature prediction of the first encoding vector can include the following steps.

[0079] The first step is to use the first prosodic prediction model to predict the prosodic features of the first encoding vector at the sentence level, and obtain the third prosodic feature vector.

[0080] In this example implementation, sentence-level prosodic feature prediction may include temporal feature processing and linear transformation of the input data of the first prosodic prediction model. For example, a recurrent neural network (e.g., GRU) can be used to process the input data temporally, and then a linear transformation is performed on the processing result. For the GRU, its input data consists of the current input data and the hidden state from the previous time step, which contains relevant information about previous nodes. Its output data consists of the output data of the hidden node at the current time step and the hidden state passed to the next node. The GRU obtains two gating states using the state transmitted from the previous node and the input data of the current node.

[0081] In this example implementation, sentence-level prosodic feature prediction can be directly performed on the output of the encoding model (the first encoding vector).

[0082] The second step involves using the second prosodic prediction model to predict the phoneme-level prosodic features of the first encoding vector, thereby obtaining the fourth prosodic feature vector.

[0083] In this example implementation, phoneme-level prosodic feature prediction may include convolutional processing and linear transformation of the input data to the second prosodic prediction model. For example, a one-dimensional convolutional neural network can be used to convolve the input data, followed by a linear transformation of the result. The convolutional processing may include multiple one-dimensional convolutions and normalization; an activation function may be added after the one-dimensional convolutions, and dropout may be applied after the normalization.

[0084] In this example implementation, phoneme-level prosodic feature prediction can be directly performed on the output of the coding model (the first coding vector). The two levels of prosodic feature prediction do not interfere with each other and are parallel.

[0085] The third step is to input the first encoding vector and the prosodic feature vector into the pre-trained variable adaptation model to perform variable prediction.

[0086] In this example implementation, the first encoding vector, the third prosodic feature vector, and the fourth prosodic feature vector can be input into a pre-trained variable adaptation model for variable prediction.

[0087] For example, refer to Figure 5 The variable adaptation model includes predictors for multiple acoustic variables, such as duration predictors, pitch predictors, and amplitude energy predictors. The outputs of the encoding model (first encoding vector), the first prosodic feature prediction model (third prosodic feature vector), and the second prosodic feature prediction model (fourth prosodic feature vector) can be input into the predictor corresponding to each acoustic variable for optimization and adjustment of timbre, prosody, etc. In some embodiments, feature overlay processing can be added after the pitch predictor and amplitude energy predictor to achieve feature overlay of pitch and amplitude energy, thereby adjusting and restoring the original timbre. The variable adaptation model also includes length regularization to adjust the output length of this module.

[0088] In this example implementation, the processing of the first encoding vector, the third prosodic feature vector, and the fourth prosodic feature vector within the predictor corresponding to each acoustic variable is as follows: Figure 6 As shown, the predictor structure can be the same for each acoustic variable. Figure 6 As shown, the pitch predictor may include a fully connected layer, two one-dimensional convolutional units and a linear layer. The one-dimensional convolutional unit consists of a one-dimensional convolutional module and a normalization module. The one-dimensional convolutional module consists of a one-dimensional convolutional layer plus a ReLU activation function, and the normalization module consists of a normalization layer plus random dropout.

[0089] For example, such as Figure 6As shown, the output of the encoding model (first encoding vector) and the output of the first prosodic feature prediction model (third prosodic feature vector) can be superimposed. Then, the superimposed features are concatenated with the output of the second prosodic feature prediction model (fourth prosodic feature vector). The concatenated result is processed sequentially through a fully connected layer, two one-dimensional convolutional units, and a linear layer to complete the optimization and adjustment of pitch and prosody.

[0090] In this example implementation, the prosody prediction model (first prosody prediction model and second prosody prediction model) is moved from the backbone network to each acoustic variable predictor of the variable adaptor, so that prosody prediction and timbre prediction are completely separated, realizing the decoupling of prosody enhancement and timbre synthesis, and achieving prosody enhancement and control effects without changing the original timbre quality.

[0091] In some embodiments, if a target object's recorded statement is available before speech synthesis, the statement to be synthesized also includes the target object's recorded statement. This recorded statement can be used as reference information and added to the variable adaptation model, making prosodic enhancement and timbre prediction more accurate. (Reference) Figure 7 or Figure 8 The method also includes the following steps.

[0092] A pre-trained first reference coding model is used to perform secondary coding on the recorded sentences to obtain a second coding vector.

[0093] In this example implementation, the first reference coding model may include multi-layer two-dimensional convolutional units, recurrent neural network units (such as GRU), multi-head attention units for pattern symbols, and linear layers. The multi-layer two-dimensional convolutional units may include two-dimensional convolutional layers, batch normalization layers, and activation functions (ReLU). The corresponding secondary coding process may include multiple rounds of two-dimensional convolution processing, batch normalization processing, activation processing, multi-head attention processing for pattern symbols, and linear processing.

[0094] The second encoded vector is input into a pre-trained variable adaptation model for variable prediction.

[0095] When the first prosodic feature vector and the second prosodic feature vector are added to the backbone network, the superposition result of the second prosodic feature vector and the superimposed feature is used to predict variables with the second encoding vector.

[0096] In this example implementation, the variable prediction process of the second encoding vector is similar to that of the prosodic feature vector. That is, the superposition result of the second encoding vector, the second prosodic feature vector, and the superposition feature is input into the variable adaptation model for corresponding processing.

[0097] When the first prosodic feature vector and the second prosodic feature vector are respectively added to the variable fitting model, the variable prediction is performed on the first encoding vector, the prosodic feature vector and the second encoding vector using the pre-trained variable fitting model.

[0098] In this example implementation, the processing of the first encoding vector, prosodic feature vector, and second encoding vector in the variable adaptation model is as follows: Figure 9 As shown, the output of the first reference coding model (second coding vector), the first coding vector, and the prosodic feature vector can be input into the predictor corresponding to each acoustic variable for optimization and adjustment of timbre, prosody, etc. The variable adaptation model also includes length regularization to adjust the output length of this module.

[0099] For example, the processing within the predictor corresponding to each acoustic variable is as follows: Figure 10 As shown, the predictor structure can be the same for each acoustic variable. Figure 10 As shown, the pitch predictor may include a fully connected layer, two one-dimensional convolutional units and a linear layer. The one-dimensional convolutional unit consists of a one-dimensional convolutional module and a normalization module. The one-dimensional convolutional module consists of a one-dimensional convolutional layer plus a ReLU activation function, and the normalization module consists of a normalization layer plus random dropout.

[0100] In this example, the output of the first reference coding model (second coding vector), the output of the first prosodic feature prediction model (third prosodic feature vector), and the first coding vector can be superimposed. Then, the superimposed features are concatenated with the output of the second prosodic feature prediction model (fourth prosodic feature vector). The concatenated result is processed sequentially through a fully connected layer, two one-dimensional convolutional units, and a linear layer to complete the optimization and adjustment of pitch and prosody.

[0101] In some embodiments, reference Figure 11 The method also includes training an acoustic prediction model. The training process can include two parts: one part is training a first reference coding model and a second reference coding model; the other part is training a first prosodic prediction model and a second prosodic prediction model, which can include the following steps S1110-S1130.

[0102] Step S1110: Obtain training samples, which include recording samples and corresponding acoustic feature samples and symbol sequence samples.

[0103] In this example implementation, the recording sample refers to a real audio recording, and the acoustic feature sample refers to the acoustic spectrum (such as a Mel spectrum) extracted from the audio recording. The audio recording has a corresponding training text, and the corresponding phoneme sequence sample is determined based on this training text. Real audio recordings generally contain speaker information (such as a speaker ID), which can be used to query a corpus to obtain the corresponding query result statement sample. The symbol sequence sample is composed of the symbol sequences corresponding to the phoneme sequence samples and the query result statement samples. A training sample may include a real audio recording, the corresponding acoustic spectrum, the corresponding phoneme sequence sample, and the symbol sequence sample corresponding to the query result statement.

[0104] In this example implementation, when training a model for a single speaker, historical audio recordings of that speaker can be obtained, and other category samples (phoneme sequences, metronomes, etc.) can be obtained from these recordings to form a training set. For a general model, historical audio recordings of many speakers can be obtained, such as recordings from groups like men, women, the elderly, and children.

[0105] Step S1120: The initial acoustic prediction model is trained once using training samples to obtain an intermediate model; the initial acoustic prediction model includes a first reference coding model and a second reference coding model.

[0106] In this example implementation, the first reference coding model can be a Global Style Tokens Reference Encoder (GST), and the second reference coding model is an acoustic feature reference encoder, such as a Mel-spectrum reference encoder. The recorded audio is input into the first reference coding model, and the corresponding Mel-spectrum is input into the second reference coding model. These two input data are used as reference information for speech synthesis. One training iteration is used to train the first reference coding model, the second reference coding model, and the backbone network.

[0107] Step S1130: Fix the model parameters of the first reference coding model and the second reference coding model of the intermediate model, and perform secondary training on the acoustic prediction model using the training samples, the first reference coding model and the second reference coding model.

[0108] In this example implementation, the model parameters of the first and second reference coding models after one training are fixed, and a second training is performed. The second training uses the first and second reference coding models to train two prosodic prediction models, namely the first prosodic prediction model (corresponding to the Mel spectral encoder) and the second prosodic prediction model (corresponding to the GST encoder). The second training is used to train the first prosodic prediction model, the second prosodic prediction model, and the backbone network.

[0109] In some embodiments, such as Figure 12 As shown, in the first part of the training process of the acoustic prediction model, two reference coding models are added, namely the first reference coding model and the second reference coding model, to form the initial acoustic prediction model. The initial acoustic prediction model is then trained once using training samples, which may include the following steps:

[0110] The first step is to use an encoding model to encode the symbol sequence samples to obtain the first sample encoding vector.

[0111] In this example implementation, the model parameters of the initial acoustic prediction model can be initialized, and the symbol sequence samples can be processed using the initialized encoding model. This may include inputting the phoneme sequence corresponding to the text sample into the phoneme embedding module for dimension mapping (embedding) to obtain an embedding vector. Alternatively, the speaker query result statement sample can be input into the speaker embedding module for dimension mapping (embedding), and the two mapping results can be superimposed to obtain the first sample encoding vector.

[0112] The second step is to encode the recording samples using the first reference coding model to obtain the second sample coding vector.

[0113] In this example implementation, the first reference coding model performs the following processing on the recording samples: multi-round two-dimensional convolution processing, batch normalization processing, activation processing, multi-head attention processing of pattern symbols, and linear processing.

[0114] The third step involves using the second reference coding model to encode the acoustic feature samples, thereby obtaining the third sample coding vector.

[0115] In this example implementation, the second reference coding model may include multiple layers of two-dimensional convolutional units, linear layers with linear normalization, multi-head attention layers, and linear layers with linear normalization. A two-dimensional convolutional unit may include a two-dimensional convolutional layer, a batch normalization layer, and an activation function (ReLU). For example, acoustic feature samples undergo multiple convolutional processes through multiple layers of two-dimensional convolutions, linear transformation and normalization, and context information extraction through a multi-head attention layer. Then, the features are subjected to linear transformation and normalization to obtain the third sample coding vector.

[0116] The fourth step involves superimposing the features of the first and second sample encoding vectors and then concatenating them with the features of the third sample encoding vector to obtain the concatenated features.

[0117] The fifth step involves using a decoding model to decode and output the spliced ​​features. Based on the output and acoustic feature samples, the first loss function value is calculated.

[0118] In this example implementation, the first loss function can be determined by the output and the mean square error (MSE) of the acoustic feature samples. The decoding process is similar to the corresponding part of the speech synthesis process for the sentence to be synthesized.

[0119] The sixth step is to adjust the model parameters of the encoding model, the first reference encoding model, the second reference encoding model, and the decoding model based on the first loss function value.

[0120] In this example implementation, the encoding model and the encoding model belong to the backbone network. The model parameters of the variable adaptation model can also be adjusted.

[0121] During a training process, the Mel spectrum corresponding to the recorded audio can be obtained. Both are used as reference information for speech synthesis, causing the GST reference encoder to output sentence-level GST encoding results and the Mel spectrum reference encoder to output phoneme-level encoding results. These, along with the encoding results from the backbone network's encoding model and the encoding results representing the speaker's timbre, are input and influence the variable adaptation model's prediction learning of each phoneme's duration, pitch, and amplitude energy. After this influence extends to the frame level, the Mel spectrum is output through the backbone network's decoding model and compared with the true Mel spectrum. The first loss function value is calculated and fed back to the network model, continuously updating the model's parameters until the first loss function value gradually converges, resulting in a reference encoding effect that achieves good performance for both the GST reference encoder and the Mel spectrum reference encoder. This process also trains other modules of the backbone network, preparing for the subsequent second part of training.

[0122] In some embodiments, such as Figure 13 As shown, in the second part of the training process, two prosodic prediction models are added to the model: a first prosodic prediction model and a second prosodic prediction model. These two prosodic prediction models are trained using the first and second reference coding models trained in the first part. In this example, the secondary training of the acoustic prediction model may include the following steps:

[0123] The first step is to use the encoded model trained once to encode the symbol sequence samples to obtain the fourth sample encoding vector.

[0124] In this example implementation, the training samples can be fed back into the model after it has been trained once.

[0125] The second step involves using the first reference coding model, trained once, to encode the recording samples and obtain the fifth sample coding vector.

[0126] The encoding process in the above two steps is the same as the corresponding part in the speech synthesis process of the sentence to be synthesized, and will not be repeated here.

[0127] The third step is to use the second reference coding model after one training to encode the acoustic feature samples and obtain the sixth sample coding vector.

[0128] In this example implementation, the second reference coding model can be a Mel-spectrum reference encoder, which is used to encode the Mel-spectrum samples using a pre-trained Mel-spectrum reference encoder. For example, the acoustic feature samples undergo multiple convolutions of multi-layer two-dimensional convolutions after one training, linear transformation and normalization, and context information extraction by a multi-head attention layer. Then, the features are subjected to linear transformation and normalization to obtain the sixth sample coding vector.

[0129] The fourth step is to superimpose the fourth sample encoding vector and the fifth sample encoding vector and then input them into the first prosody prediction model and the second prosody prediction model, respectively.

[0130] Fifth, calculate the second loss function value based on the fifth sample encoding vector and the output of the first prosody prediction model; adjust the model parameters of the first prosody prediction model based on the second loss function value.

[0131] The sixth step is to calculate the third loss function value based on the sixth sample encoding vector and the output of the second prosody prediction model; and to adjust the model parameters of the second prosody prediction model based on the third loss function value.

[0132] This example may also include the following steps: the sixth sample encoding vector is concatenated with the backbone network and then processed by the variable adaptation model and the decoding model. The output result is compared with the real acoustic features (acoustic feature samples) to calculate the loss function and adjust the model parameters of the backbone network.

[0133] In the above embodiments, the second loss function value and the third loss function value can also be determined using the mean squared error (MSE). For example... Figure 13As shown, during the secondary training process, the model parameters of the GST reference encoder and the Mel spectrum reference encoder are fixed so that they are not updated during this part of the training. A first prosodic prediction model (sentence-level prosodic predictor) and a second prosodic prediction model (phoneme-level prosodic predictor) are added to the initial acoustic model from the first part of the training. During the secondary training, the sentences and Mel spectra of the corresponding text are used as reference information, causing the GST reference encoder and the Mel spectrum reference encoder to output their respective encoding results. Simultaneously, the sentence-level prosodic predictor and the phoneme-level prosodic predictor output their prediction results. By comparing the GST encoding results with the prediction results of the sentence-level prosodic predictor, the model parameters of the sentence-level prosodic predictor are adjusted; similarly, by comparing the prediction results of the Mel spectrum reference encoder and the phoneme-level prosodic predictor, the model parameters of the phoneme-level prosodic predictor are adjusted. Additionally, the encoding results of the Mel spectrum reference encoder can be input into the backbone network for feature concatenation, then processed through a fully connected layer before being input into the variable adaptation model and the decoding model for further processing, outputting predicted acoustic features. The first loss function value can be calculated using the predicted acoustic features and acoustic feature samples. This first loss function value is then used to adjust the model parameters of the backbone network and the variable adaptation model. This process is repeated to continuously update the parameters of the acoustic prediction model until all loss function values ​​converge, at which point training ends.

[0134] Understandably, the acoustic prediction model disclosed herein can be trained first, and then the trained model can be used for the speech synthesis process of the sentence to be synthesized.

[0135] This disclosure studies the specific requirements of prosody in particular scenarios. Prosody typically reflects the characteristics of a speaker's voice, such as pitch, volume, pace, and pauses. Due to differences in prosody, even the same words can evoke different emotions in the listener. For example, the phrase "Hello," if pronounced loudly and fully with a higher pitch in the latter half, will convey a warm and cheerful feeling to the listener. Conversely, if pronounced quickly with a lower pitch in the latter half, it will convey a cold or indifferent feeling. Therefore, prosody plays a crucial role in the information conveyed by speech.

[0136] While existing speech synthesis systems can output high-definition audio with high timbre similarity, in specific contexts or business scenarios, the synthesized results tend to be relatively fixed after the model converges during training. The synthesized audio lacks the expressiveness to capture the specific style or rhythm required by the business scenario, resulting in low-quality speech that fails to replicate the effect of a real speaker in that context. These specific business scenarios may include situations where a rising intonation or rising tone at the end of sentences is desired, a more rhythmic delivery is desired, or features of colloquialisms or live-streaming sales styles are added. Because existing models are trained using a large amount of timbre data from different speakers, and the consistency of audio recordings from the same speaker cannot be guaranteed, the rhythmic quality of the output tends to be averaged out.

[0137] To improve the model's performance in terms of prosody and style, this disclosure introduces a prosodic feature prediction model to guide the audio synthesis process, thereby achieving better prosodic control and enhancing prosodic effects. Specifically, this disclosure introduces sentence-level and factor-level prosodic predictors to guide the learning of sentence-level and phoneme-level prosodic features, enabling better reproduction of the speaker's prosodic characteristics and improving the intonation and naturalness of the synthesized audio. In complex prosodic modeling scenarios involving speakers with passionate or emotionally charged delivery, this approach achieves prosodic enhancement, improving the quality of the synthesized speech.

[0138] Because prosodic style reflects the pitch, volume, speed, rhythm, and pauses of a speaker's voice, and is completely unrelated to the speaker's timbre—meaning everyone can use their own timbre to learn the intonation style of any speaker—mixing predicted prosodic features into the backbone network may alter the timbre of some synthesized audio. Furthermore, considering the close relationship between prosodic style features and the variable fitting model's prediction of the duration, pitch, and amplitude energy of each phoneme, this disclosure inputs the prediction results of the prosodic feature prediction model into the variable fitting model, without mixing them with the timbre features of the backbone network. This avoids the influence of prosodic style features on the speaker's timbre, decoupling timbre from prosodic style, ensuring that the synthesized audio only learns the prosodic style features of the reference information and completely eliminates the influence on the original speaker's timbre.

[0139] Within the variable adaptation model, the encoding result of the GST reference encoder is independent of the intermediate results on the backbone network and only affects the prediction of prosodic style-related duration, pitch, and amplitude energy. This allows the model to learn only the prosodic style from the reference information, thus decoupling timbre from prosodic style and achieving better prosodic control through the reference information.

[0140] This disclosure uses the prediction results of the prosody prediction model as input variables to adapt the model, avoiding the influence of the prediction results on the backbone network, so that the prosody enhancement process of the prosody prediction model is completely separated from the timbre prediction, and at the same time achieves the effects of prosody enhancement and control.

[0141] In the model training phase, this disclosure uses two reference encoders to guide the learning of prosodic features at the sentence and phoneme levels. In the speech synthesis phase, it can independently complete speech synthesis without using reference encoders and reference recording information (considering that in TTS speech synthesis, there is usually only text information and no recorded audio). At the same time, it achieves prosodic enhancement and control, increasing practicality.

[0142] This disclosure designs the addition position of the GST reference encoder (first reference coding model), adds it to the variable adaptation model, so that it only serves as an optimization adjustment for the prosodic style and has no effect on the synthesized timbre, thereby achieving decoupling of timbre and prosodic style.

[0143] This disclosure can be applied to speech-related business scenarios that have high requirements for rhythm and style in different business scenarios such as human-computer interaction and novel reading.

[0144] Furthermore, in this exemplary embodiment, a speech synthesis device 1400 is also provided. This speech synthesis device 1400 can be applied to intelligent voice devices. (Reference) Figure 14 As shown, the speech synthesis device 1400 may include: an acquisition module 1410, a prediction module 1420, and a speech synthesis module 1430. The acquisition module 1410 is used to acquire the symbol sequence of the sentence to be synthesized, which includes the text to be synthesized and the query result statement for the target object. The prediction module 1420 is used to predict the acoustic features of the symbol sequence using a pre-trained acoustic prediction model to obtain the acoustic features corresponding to the sentence to be synthesized. The acoustic prediction model includes a prosody prediction model, which learns the prosody features of the reference audio recording during the model training stage to enhance the prosody features of the sentence to be synthesized during the speech synthesis stage. The speech synthesis module 1430 is used to perform feature conversion and synthesis on the acoustic features to obtain the speech corresponding to the sentence to be synthesized.

[0145] In one exemplary embodiment of this disclosure, the acoustic prediction model further includes an encoding model and a decoding model. The prediction module 1420 can also be used to: use the pre-trained encoding model to encode the symbol sequence once to obtain a first encoding vector; use the pre-trained prosody prediction model to predict the prosody features of the first encoding vector to obtain a prosody feature vector; and predict the acoustic features of the sentence to be synthesized based on the pre-trained decoding model, the first encoding vector, and the prosody feature vector to obtain the acoustic features corresponding to the sentence to be synthesized.

[0146] In an exemplary embodiment of this disclosure, the prediction module 1420 may also be used to: use a pre-trained variable adaptation model to perform variable prediction on the superposition result of the first encoding vector and the prosodic feature vector to obtain a first variable prediction result; and use a pre-trained decoding model to perform attention-based decoding processing on the first variable prediction result to obtain the acoustic features corresponding to the sentence to be synthesized.

[0147] In an exemplary embodiment of this disclosure, the prediction module 1420 may also be used to: input the first encoding vector and the prosodic feature vector into a pre-trained variable adaptation model respectively, perform variable prediction, and obtain a second variable prediction result; and use a pre-trained decoding model to perform attention-based decoding processing on the second variable prediction result to obtain the acoustic features corresponding to the sentence to be synthesized.

[0148] In an exemplary embodiment of this disclosure, the prosody prediction model includes a first prosody prediction model and a second prosody prediction model. The prediction module 1420 can also be used to: use the first prosody prediction model to perform sentence-level prosody feature prediction on the first encoding vector to obtain a first prosody feature vector; use the second prosody prediction model to perform phoneme-level prosody feature prediction on the superimposed features of the first encoding vector and the first prosody feature vector to obtain a second prosody feature vector; and perform variable prediction on the superimposed result of the second prosody feature vector and the superimposed features.

[0149] In one exemplary embodiment of this disclosure, the prosody prediction model includes a first prosody prediction model and a second prosody prediction model. The prediction module 1420 can also be used to: use the first prosody prediction model to perform sentence-level prosody feature prediction on the first encoding vector to obtain a third prosody feature vector; use the second prosody prediction model to perform phoneme-level prosody feature prediction on the first encoding vector to obtain a fourth prosody feature vector; and input the first encoding vector, the third prosody feature vector, and the fourth prosody feature vector into a pre-trained variable adaptation model to perform variable prediction.

[0150] In one exemplary embodiment of this disclosure, the prediction module 1420 may also be used to perform temporal feature processing and linear transformation on the input data of the first prosody prediction model.

[0151] In one exemplary embodiment of this disclosure, the prediction module 1420 may also be used to: perform convolution processing and linear transformation on the input data of the second prosody prediction model.

[0152] In an exemplary embodiment of this disclosure, the sentence to be synthesized further includes the recorded sentence of the target object, and the apparatus 1400 further includes a reference module, which is used to perform secondary encoding processing on the recorded sentence using a pre-trained first reference encoding model to obtain a second encoding vector; the prediction module 1420 can also be used to: use a pre-trained variable adaptation model to perform variable prediction on the superposition result of the second prosodic feature vector and the superposition feature, and the second encoding vector; or, use a pre-trained variable adaptation model to perform variable prediction on the first encoding vector, the prosodic feature vector and the second encoding vector.

[0153] In one exemplary embodiment of this disclosure, the apparatus 1400 further includes a training module, which can be used to train an acoustic prediction model. The training module includes: an acquisition submodule, a first training submodule, and a second training submodule. The acquisition submodule can be used to acquire training samples, which include recording samples and corresponding acoustic feature samples and symbol sequence samples. The first training submodule can be used to train an initial acoustic prediction model once using the training samples to obtain an intermediate model. The initial acoustic prediction model includes a first reference coding model and a second reference coding model. The second training submodule can be used to fix the model parameters of the first reference coding model and the second reference coding model of the intermediate model, and to perform a second training on the acoustic prediction model using the training samples, the first reference coding model, and the second reference coding model.

[0154] In one exemplary embodiment of this disclosure, the initial acoustic prediction model further includes an encoding model and a decoding model. The first training submodule can also be used to encode symbol sequence samples using the encoding model to obtain a first sample encoding vector; to encode recording samples using a first reference encoding model to obtain a second sample encoding vector; to encode acoustic feature samples using a second reference encoding model to obtain a third sample encoding vector; to superimpose features on the first and second sample encoding vectors and then concatenate them with the third sample encoding vector to obtain concatenated features; to decode the concatenated features using the decoding model and output them; to calculate a first loss function value based on the output result and the acoustic feature samples; and to adjust the model parameters of the encoding model, the first reference encoding model, the second reference encoding model, and the decoding model based on the first loss function value.

[0155] In one exemplary embodiment of this disclosure, the acoustic prediction model includes a first prosody prediction model and a second prosody prediction model. The second training submodule can be used to: encode symbol sequence samples using a once-trained encoding model to obtain a fourth sample encoding vector; encode recording samples using a once-trained first reference encoding model to obtain a fifth sample encoding vector; superimpose the fourth and fifth sample encoding vectors and input them into the first and second prosody prediction models respectively; calculate a second loss function value based on the fifth sample encoding vector and the output of the first prosody prediction model; adjust the model parameters of the first prosody prediction model based on the second loss function value; calculate a third loss function value based on the fifth sample encoding vector and the output of the second prosody prediction model; and adjust the model parameters of the second prosody prediction model based on the third loss function value.

[0156] The specific details of each module or unit in the aforementioned speech synthesis device have been described in detail in the corresponding speech synthesis methods, so they will not be repeated here.

[0157] On the other hand, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the methods as described in the following embodiments. For example, the electronic device may perform... Figures 2 to 13 The various steps shown are as follows.

[0158] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0159] The following reference Figure 15 To describe an electronic device 1500 according to such an embodiment of the present disclosure. Figure 15 The electronic device 1500 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0160] like Figure 15 As shown, the electronic device 1500 is manifested in the form of a general-purpose computing device. The components of the electronic device 1500 may include, but are not limited to: at least one processing unit 1510, at least one storage unit 1520, a bus 1530 connecting different system components (including storage unit 1520 and processing unit 1510), and a display unit 1540.

[0161] The storage unit stores program code, which can be executed by the processing unit 1510, causing the processing unit 1510 to perform the steps described in the "Exemplary Methods" section above according to various exemplary embodiments of this disclosure.

[0162] For example, the processing unit 1510 may perform the following steps: obtaining a symbol sequence of a sentence to be synthesized, the sentence to be synthesized including the text to be synthesized and a query result statement for the target object; using a pre-trained acoustic prediction model to predict the acoustic features of the symbol sequence to obtain the acoustic features corresponding to the sentence to be synthesized; the acoustic prediction model includes a prosody prediction model, which learns the prosody features of a reference audio recording during the model training phase to enhance the prosody features of the sentence to be synthesized during the speech synthesis phase; and performing feature transformation and synthesis on the acoustic features to obtain the speech corresponding to the sentence to be synthesized.

[0163] Storage unit 1520 may include readable media in the form of volatile storage units, such as random access memory (RAM) 15201 and / or cache memory 15202, and may further include read-only memory (ROM) 15203.

[0164] Storage unit 1520 may also include a program / utility 15204 having a set (at least one) program module 15205, such program module 15205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0165] Bus 1530 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0166] Electronic device 1500 can also communicate with one or more external devices 1570 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 1500, and / or with any device that enables electronic device 1500 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1550. Furthermore, electronic device 1500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1560. As shown, network adapter 1560 communicates with other modules of electronic device 1500 via bus 1530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RA identification systems, tape drives, and data backup storage systems.

[0167] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0168] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0169] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps, such as omitting certain steps, combining multiple steps into one step, and / or breaking down one step into multiple steps, should all be considered part of this disclosure.

[0170] It should be understood that this disclosure, as disclosed and defined herein, extends to all alternative combinations of two or more individual features mentioned or apparent in the text and / or figures. All these different combinations constitute multiple alternative aspects of this disclosure. The embodiments described in this specification illustrate the best known mode for implementing this disclosure and will enable those skilled in the art to utilize it.

Claims

1. A speech synthesis method, characterized in that, The method includes: Obtain the symbol sequence of the statement to be synthesized, which includes the text to be synthesized and the query result statement for the target object; Using a pre-trained acoustic prediction model, acoustic features are predicted on the symbol sequence to obtain the acoustic features corresponding to the sentence to be synthesized. The acoustic prediction model includes an encoding model, a prosody prediction model, and a decoding model. The prosody prediction model learns the prosody features of a reference audio recording during the model training phase to enhance the prosody features of the sentence to be synthesized during the speech synthesis phase. The acoustic features are transformed and synthesized to obtain the speech corresponding to the sentence to be synthesized; The acoustic features corresponding to the sentence to be synthesized are generated as follows: The symbol sequence is encoded once using a pre-trained encoding model to obtain a first encoding vector; a prosodic prediction model is used to predict the prosodic features of the first encoding vector to obtain a prosodic feature vector; and the acoustic features of the sentence to be synthesized are predicted based on a pre-trained decoding model, the first encoding vector, and the prosodic feature vector to obtain the acoustic features corresponding to the sentence to be synthesized. This includes: using a pre-trained variable adaptation model to predict variables in the superposition result of the first encoding vector and the prosodic feature vector to obtain a first variable prediction result; and using a pre-trained decoding model to perform attention-based decoding on the first variable prediction result to obtain the acoustic features corresponding to the sentence to be synthesized.

2. The speech synthesis method according to claim 1, characterized in that, The step of predicting the acoustic features of the sentence to be synthesized based on the pre-trained decoding model, the first encoding vector, and the prosodic feature vector includes: The first encoding vector and the prosodic feature vector are respectively input into a pre-trained variable adaptation model to perform variable prediction and obtain the second variable prediction result. Using a pre-trained decoding model, the prediction result of the second variable is decoded based on an attention mechanism to obtain the acoustic features corresponding to the sentence to be synthesized.

3. The speech synthesis method according to claim 1, characterized in that, The prosodic prediction model includes a first prosodic prediction model and a second prosodic prediction model. The step of using the pre-trained prosodic prediction model to predict prosodic features on the first encoded vector includes: Using the first prosody prediction model, sentence-level prosody feature prediction is performed on the first encoding vector to obtain the first prosody feature vector; Using the second prosodic prediction model, phoneme-level prosodic feature prediction is performed on the superimposed features of the first encoding vector and the first prosodic feature vector to obtain the second prosodic feature vector. The variable prediction based on the superposition result of the first encoding vector and the prosodic feature vector includes: Variable prediction is performed on the superposition result of the second prosodic feature vector and the superimposed feature.

4. The speech synthesis method according to claim 2, characterized in that, The prosodic prediction model includes a first prosodic prediction model and a second prosodic prediction model. The step of using the pre-trained prosodic prediction model to predict prosodic features on the first encoded vector includes: Using the first prosody prediction model, sentence-level prosody feature prediction is performed on the first encoding vector to obtain the third prosody feature vector; Using the second prosodic prediction model, phoneme-level prosodic feature prediction is performed on the first encoding vector to obtain the fourth prosodic feature vector; The step of inputting the first encoded vector and the prosodic feature vector into a pre-trained variable adaptation model for variable prediction includes: The first encoding vector, the third prosodic feature vector, and the fourth prosodic feature vector are respectively input into a pre-trained variable adaptation model for variable prediction.

5. The speech synthesis method according to claim 3 or 4, characterized in that, The sentence-level prosodic feature prediction includes: The input data of the first prosody prediction model is subjected to temporal feature processing and linear transformation.

6. The speech synthesis method according to claim 3 or 4, characterized in that, The phoneme-level prosodic feature prediction includes: The input data of the second prosody prediction model is subjected to convolution processing and linear transformation.

7. The speech synthesis method according to claim 3, characterized in that, The statement to be synthesized also includes the recorded statement of the target object, and the method further includes: The recorded statement is encoded twice using a pre-trained first reference coding model to obtain a second coding vector. The variable prediction includes: Using a pre-trained variable adaptation model, the superposition result of the second prosodic feature vector and the superimposed features is compared with the second encoding vector to predict variables; or, Using a pre-trained variable adaptation model, variable prediction is performed on the first encoding vector, the prosodic feature vector, and the second encoding vector.

8. The speech synthesis method according to claim 1, characterized in that, The method further includes training the acoustic prediction model, the training process comprising: Acquire training samples, which include recording samples and corresponding acoustic feature samples and symbol sequence samples; The initial acoustic prediction model is trained once using the training samples to obtain an intermediate model; the initial acoustic prediction model includes a first reference coding model and a second reference coding model. The model parameters of the first reference coding model and the second reference coding model of the intermediate model are fixed, and the acoustic prediction model is trained a second time using the training samples, the first reference coding model and the second reference coding model.

9. The speech synthesis method according to claim 8, characterized in that, The initial acoustic prediction model further includes an encoding model and a decoding model. The step of training the initial acoustic prediction model once using the training samples includes: The symbol sequence samples are encoded using an encoding model to obtain the first sample encoding vector; The recording samples are encoded using the first reference coding model to obtain the second sample coding vector; The acoustic feature samples are encoded using a second reference coding model to obtain the third sample coding vector; After superimposing features on the first sample encoding vector and the second sample encoding vector, features are concatenated with the third sample encoding vector to obtain concatenated features; The spliced ​​features are decoded using a decoding model and output. The first loss function value is calculated based on the output result and the acoustic feature sample. Based on the first loss function value, adjust the model parameters of the encoding model, the first reference encoding model, the second reference encoding model, and the decoding model.

10. The speech synthesis method according to claim 9, characterized in that, The acoustic prediction model includes a first prosodic prediction model and a second prosodic prediction model. The secondary training of the acoustic prediction model includes: The symbol sequence samples are encoded using the encoding model trained once to obtain the fourth sample encoding vector; The first reference coding model after one training is used to encode the recording samples to obtain the fifth sample coding vector; The acoustic feature samples are encoded using the second reference coding model after one training to obtain the sixth sample coding vector; The fourth sample encoding vector and the fifth sample encoding vector are superimposed and then input into the first prosody prediction model and the second prosody prediction model, respectively. Based on the fifth sample encoding vector and the output of the first prosody prediction model, calculate the second loss function value; and adjust the model parameters of the first prosody prediction model based on the second loss function value. Based on the sixth sample encoding vector and the output of the second prosody prediction model, the third loss function value is calculated; based on the third loss function value, the model parameters of the second prosody prediction model are adjusted.

11. A speech synthesis device, characterized in that, The device includes: The acquisition module is used to acquire the symbol sequence of the statement to be synthesized, which includes the text to be synthesized and the query result statement for the target object; The prediction module is used to predict the acoustic features of the symbol sequence using a pre-trained acoustic prediction model to obtain the acoustic features corresponding to the sentence to be synthesized. The acoustic prediction model includes an encoding model, a prosody prediction model and a decoding model. The prosody prediction model learns the prosody features of the reference audio recording during the model training stage to enhance the prosody features of the sentence to be synthesized during the speech synthesis stage. The speech synthesis module is used to perform feature conversion and synthesis on the acoustic features to obtain the speech corresponding to the sentence to be synthesized; The acoustic features corresponding to the sentence to be synthesized are generated as follows: The symbol sequence is encoded once using a pre-trained encoding model to obtain a first encoding vector; a prosodic prediction model is used to predict the prosodic features of the first encoding vector to obtain a prosodic feature vector; and the acoustic features of the sentence to be synthesized are predicted based on a pre-trained decoding model, the first encoding vector, and the prosodic feature vector to obtain the acoustic features corresponding to the sentence to be synthesized. This includes: using a pre-trained variable adaptation model to predict variables in the superposition result of the first encoding vector and the prosodic feature vector to obtain a first variable prediction result; and using a pre-trained decoding model to perform attention-based decoding on the first variable prediction result to obtain the acoustic features corresponding to the sentence to be synthesized.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

13. An electronic device, characterized in that, include: One or more processors; as well as A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1-10.

Images