Model training method and device, speech synthesis method, equipment and storage medium

Abstract

Embodiments of the present application provide a model training method and device, a speech synthesis method, equipment and a storage medium, and relate to the technical field of artificial intelligence. The model training method obtains a training data set, obtains a monotonic alignment loss function used for training an attention unit and a preset loss function used for training a speech synthesis model, trains the speech synthesis model based on the preset loss function in combination with a speech output vector and a corresponding speech label, adjusts model parameters of the speech synthesis model, and obtains a trained speech synthesis model when a value of the loss function meets a preset condition. In the training process, the monotonicity of an attention weight sequence is trained based on the monotonic alignment loss function. The monotonic alignment loss function is set for attention alignment, the monotonicity of the attention weight sequence is ensured, which helps to realize fast convergence of the model, improve the training accuracy of the model, and improve the naturalness and robustness of the synthesized language.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to model training methods and apparatus, speech synthesis methods, devices and storage media. Background Technology

[0002] Speech synthesis is the process of generating corresponding speech information from text. It is crucial for improving the realism of human-computer interaction systems. For example, speech synthesis technology is used in common scenarios such as voice assistants, virtual humans, and pronunciation aids.

[0003] Currently, deep learning can be used for speech synthesis. Deep learning-based speech synthesis can generate natural speech close to human level, especially autoregressive speech synthesis models. This involves constructing an end-to-end neural network model to achieve sequence mapping between text and speech melspectral data. The mainstream method for speech synthesis currently uses an Encoder-Decoder framework to build the speech synthesis model. This model's attention alignment is used to establish the alignment relationship between the text sequence and the melspectral sequence. However, related techniques either use a diagonalization method based on a Laplace distribution for the alignment weight matrix, or iteratively calculate the optimal alignment relationship using the Viterbi algorithm. Neither of these methods can achieve precise alignment, which may cause problems such as difficulty in model training convergence, affecting model training stability, or inaccurate speech synthesis results leading to word repetition or omission. Summary of the Invention

[0004] The main objective of this invention is to propose a model training method and apparatus, a speech synthesis method, a device, and a storage medium, and to select an appropriate attention alignment mechanism to improve the model convergence speed and the accuracy of speech synthesis results.

[0005] To achieve the above objectives, a first aspect of this invention proposes a model training method for training a speech synthesis model, the speech synthesis model comprising an encoding unit, an attention unit, and a decoding unit, including:

[0006] Obtain a training dataset, which includes text sequence samples and speech labels for the text sequence samples;

[0007] The text sequence sample is subjected to feature extraction by the encoding unit to obtain a latent feature vector;

[0008] The attention unit selects attention from the latent feature vector to obtain an attention weight sequence;

[0009] The latent feature vector and the attention weight sequence are decoded by the decoding unit to obtain the speech output vector;

[0010] The speech output vector and the speech label are trained by a preset loss function, the model parameters of the speech synthesis model are adjusted, and the attention weight sequence is monotonically trained based on the monotonic alignment loss function until the value of the loss function converges, thus obtaining the trained speech synthesis model.

[0011] In one embodiment, the step of decoding the latent feature vector and the attention weight sequence through the decoding unit to obtain the speech output vector includes:

[0012] Generate a context vector based on the latent feature vector and the attention weight sequence;

[0013] Obtain the state vector generated by the decoding unit of the speech synthesis model;

[0014] A speech output vector is generated based on the state vector and the context vector.

[0015] In one embodiment, the step of performing attention selection on the latent feature vector through the attention unit to obtain an attention weight sequence includes:

[0016] The attention unit performs attention selection on the latent feature vector for each frame of speech in the speech output vector to obtain the attention weight sequence of speech for each frame of speech in the speech output vector.

[0017] The step of monotonically training the attention weight sequence based on the monotonic alignment loss function includes:

[0018] Calculate the attention weights of the speech in the current frame based on the attention weight sequence of the speech in the current frame;

[0019] Calculate the attention weights of the speech in the next frame based on the attention weight sequence of the speech in the next frame;

[0020] Calculate the difference between the attention weight of the speech in the current frame and the attention weight of the speech in the next frame to obtain the attention weight difference;

[0021] The function value of the monotonic alignment loss function is calculated based on the attention weight difference;

[0022] The attention weight sequence is adjusted according to the function value until the value of the monotonic alignment loss function meets a preset condition.

[0023] In one embodiment, calculating the function value of the monotonic alignment loss function based on the attention weight difference includes:

[0024] The difference is input into a preset activation function for calculation to obtain the activation value;

[0025] The function value of the monotonic alignment loss function is obtained by summing the activation values ​​of the speech of all frames in the speech output vector.

[0026] Wherein, the activation function is the ReLU function, and the monotonic alignment loss function is expressed as:

[0027]

[0028] Where Loss1 represents the monotonic alignment loss function, T represents the total number of frames in the speech output vector, and a i The attention weight sequence represents the i-th frame of speech. The attention weight, a, represents the attention weight of the i-th frame of speech. i+1 The attention weight sequence represents the (i+1)th frame of speech. The attention weight represents the speech in the (i+1)th frame.

[0029] In one embodiment, the step of calculating the attention weight of the speech in the current frame based on the attention weight sequence of the speech in the current frame is expressed as follows:

[0030] Calculate the target location of the attention weight sequence of the speech in the current frame;

[0031] Calculate the probability that the speech in the current frame is the text at the target location in the text sequence based on the target location;

[0032] The attention weight of the speech in the current frame is calculated based on the target location and the probability.

[0033] Represented as:

[0034]

[0035]

[0036] Where L represents the number of texts in the text sequence, j represents the target position, and a i,j This represents the probability that the speech in the i-th frame is the text at the j-th position in the text sequence.

[0037] In one embodiment, generating a speech output vector based on the state vector and the context vector includes:

[0038] Generate Mel speech vectors based on the state vector and the context vector;

[0039] The speech output vector is generated based on the Mel speech vector.

[0040] To achieve the above objectives, a second aspect of the present invention provides a speech synthesis method, comprising:

[0041] Obtain the text sequence to be synthesized;

[0042] The text sequence to be synthesized is input into a speech synthesis model for speech synthesis processing to obtain a speech output sequence. The speech synthesis model is trained using the model training method described in any of the first aspects.

[0043] To achieve the above objectives, a third aspect of the present invention provides a training apparatus for training a speech synthesis model, the speech synthesis model comprising an encoding unit, an attention unit, and a decoding unit, including:

[0044] The training data acquisition module is used to acquire a training dataset, which includes text sequence samples and speech tags for the text sequence samples.

[0045] An encoding module is used to extract features from the text sequence sample through the encoding unit to obtain a latent feature vector;

[0046] The attention module is used to select attention from the latent feature vector through the attention unit to obtain an attention weight sequence;

[0047] The decoding module is used to decode the latent feature vector and the attention weight sequence through the decoding unit to obtain the speech output vector;

[0048] The training module is used to train the speech output vector and the speech label using a preset loss function, adjust the model parameters of the speech synthesis model, and perform monotonic training on the attention weight sequence based on the monotonic alignment loss function until the value of the loss function converges, thereby obtaining the trained speech synthesis model.

[0049] To achieve the above objectives, a fourth aspect of the present invention provides an electronic device comprising:

[0050] At least one memory;

[0051] At least one processor;

[0052] At least one program;

[0053] The program is stored in a memory, and the processor executes the at least one program to implement the method described in the first or second aspect of the present invention.

[0054] To achieve the above objectives, a fifth aspect of the present invention provides a storage medium that is a computer-readable storage medium storing computer-executable instructions for causing a computer to perform:

[0055] The method described in the first or second aspect above.

[0056] The speech synthesis method, model training method, apparatus, device, and storage medium proposed in this invention acquire a training dataset, obtain a monotonic alignment loss function for training the attention unit, and a preset loss function for training the speech synthesis model. Using an encoding unit, latent feature vectors are obtained based on text sequence samples. Using an attention unit, an attention weight sequence is generated. Using a decoding unit, a speech output vector is obtained based on the latent feature vector and the attention weight sequence. Finally, based on the preset loss function, the speech synthesis model is trained using the speech output vector and corresponding speech tags. The model parameters are adjusted until the value of the loss function meets preset conditions, resulting in a trained speech synthesis model. During training, the attention weight sequence is trained monotonically based on the monotonic alignment loss function. This embodiment sets a monotonic alignment loss function for attention alignment to ensure the monotonicity of the attention weight sequence, which helps achieve rapid model convergence, improves model training accuracy, avoids word repetition or omission caused by inaccurate speech synthesis results, and enhances the naturalness and robustness of the synthesized language. Attached Figure Description

[0057] Figure 1 This is a flowchart of the model training method provided in the embodiments of the present invention.

[0058] Figure 2 This is a flowchart of a model training method provided in another embodiment of the present invention.

[0059] Figure 3 This is a flowchart of a model training method provided in another embodiment of the present invention.

[0060] Figure 4 This is a flowchart of a model training method provided in another embodiment of the present invention.

[0061] Figure 5 This is a flowchart of a model training method provided in another embodiment of the present invention.

[0062] Figure 6 This is a schematic diagram of the attention mechanism of a model training method provided in another embodiment of the present invention.

[0063] Figure 7 This is a flowchart of the speech synthesis method provided in the embodiments of the present invention.

[0064] Figure 8 This is a structural block diagram of a model training device provided in another embodiment of the present invention.

[0065] Figure 9 This is a structural block diagram of a speech synthesis device provided in another embodiment of the present invention.

[0066] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0068] It should be noted that although functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0070] First, let's clarify some of the terms used in this invention:

[0071] Speech technology refers to key technologies in the computer field, including Automatic Speech Recognition (ASR) and Text-to-Speech (TTS). Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, and speech has become one of the most promising human-computer interaction methods, offering several advantages over other interaction methods.

[0072] Attention units: Research on attention units first appeared in psychology, where it was observed that when a person sees a scene, although they are faced with the entire scene, their attention is focused on only a specific point. In 2014, Bahdanau et al. first introduced attention units into machine translation, combining them with neural networks to improve the accuracy of machine translation. Subsequent studies have attempted to combine attention units with neural networks for different tasks, all achieving significant advantages over traditional methods to varying degrees.

[0073] Deep learning involves learning the inherent patterns and hierarchical representations of sample data. The information gained during this learning process greatly aids in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to possess analytical and learning capabilities similar to humans, allowing them to recognize data such as text, images, and sound. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition far exceeding previous related technologies. Deep learning has also yielded significant advancements in search technology, data mining, machine learning, machine translation, natural language processing, multimedia learning, speech recognition, recommendation and personalization technologies, and other related fields. Deep learning enables machines to mimic human activities such as sight, hearing, and thought, solving many complex pattern recognition problems and significantly advancing artificial intelligence-related technologies.

[0074] Recurrent Neural Networks (RNNs): Compared to deep neural networks, the most fundamental difference of RNNs is that the input not only includes the current speech features but also the internal state. The internal state encodes information from past time series processed by the RNN. Unlike deep neural networks, which statically transform input into output, RNNs are dynamic systems capable of representing and learning correlations between sequences over long periods. The speech recognition in this application is a sequence learning task, and the structural characteristics of RNNs give them a strong ability to model serialized data, thus offering significant advantages in speech recognition.

[0075] Mel spectrogram: This is the spectrum obtained by performing a Fourier transform on an acoustic signal, followed by a Mel-scale transformation. Spectrograms are often very large images; to obtain appropriately sized sound features, they can be transformed into Mel spectrograms after passing through a Mel-scale filter bank. In the Mel frequency domain, there is a linear relationship between the Mel frequencies of speech and human pitch perception. The Mel spectrum is obtained by combining the Mel frequency cepstrum and the spectrogram.

[0076] Speech synthesis is one of the key technologies in human-computer interaction systems. As an important research direction in the field of speech, speech synthesis technology has broad application prospects in areas such as intelligent navigation, robotics, intelligent reading, and smart tourism. For example, common scenarios such as voice assistants, virtual humans, and pronunciation aids all utilize speech synthesis technology. Speech synthesis generates corresponding speech information from text, which is crucial for improving the realism of human-computer interaction systems. Some voice assistants, such as Siri (Apple), Xiaodu (Baidu), Xiaoyi (Huawei), and Xiao Ai (Xiaomi), have enabled the widespread application of speech synthesis in scenarios such as search, navigation, and human-computer dialogue. Speech synthesis is also used for announcements in public places such as airports, train stations, and hospitals; for calls on electronic devices such as mobile phones and laptops; for in-vehicle navigation systems; and for language learning applications.

[0077] Currently, deep learning can be used for speech synthesis. Deep learning-based speech synthesis can generate natural speech close to human level, especially autoregressive speech synthesis models. This involves constructing an end-to-end neural network model to achieve sequence mapping between text and speech melspectral sequences. The mainstream approach in the speech field currently combines an Encoder-Decoder framework with attention units to build speech synthesis models. This model uses attention alignment to establish the alignment relationship between the text sequence and the melspectral sequence. The output speech quality of a speech synthesis model can be judged based on two main features: first, the clarity and intelligibility of the speech content; and second, the degree to which the generated sound resembles human language. If the attention alignment is not precise, it will affect the output speech quality.

[0078] Related technologies either model alignment by using a diagonalization method where the alignment weight matrix follows a Laplace distribution, or iteratively calculate the optimal alignment relationship using the Viterbi algorithm. Neither of these methods can achieve precise alignment, potentially causing problems such as difficulty in model training convergence, affecting model training stability, or resulting in inaccurate speech synthesis results leading to word repetition or omission.

[0079] Based on this, embodiments of the present invention provide a speech synthesis method, a model training method, an apparatus, a device, and a storage medium. By setting a monotonic alignment loss function for attention alignment, the monotonicity of the attention weight sequence is ensured, which helps to achieve rapid model convergence. At the same time, it can improve the accuracy of model training, avoid the problem of word repetition or omission caused by inaccurate speech synthesis results, and improve the naturalness and robustness of synthesized language.

[0080] The present invention provides a speech synthesis method, a model training method, an apparatus, a device, and a storage medium, which are specifically described through the following embodiments. First, the model training method in the embodiments of the present invention is described.

[0081] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0082] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0083] The speech synthesis method and model training method provided in this invention relate to the field of artificial intelligence technology, and particularly to the field of data mining technology. The speech synthesis method and model training method provided in this invention can be applied to a terminal, a server, or software running on either a terminal or a server. The terminal communicates with the server via a network. The speech synthesis method and model training method can be executed by the terminal or the server, or jointly by both. Here, we illustrate the model training method using a terminal as an example: The terminal acquires a training dataset and determines a loss function for training the speech synthesis model. The loss function includes a monotonic alignment loss function and a preset loss function. The monotonic alignment loss function characterizes the monotonicity of the attention weight sequence generated by the speech synthesis model, and the preset loss function characterizes the difference between the speech output sequence and the speech labels of the speech synthesis model. Finally, the speech synthesis model is trained based on the loss function and the training dataset until the value of the loss function meets a preset condition, resulting in a trained speech synthesis model.

[0084] In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, or smartwatch, etc. Furthermore, the terminal can also be a smart in-vehicle device that can use text sequences for speech synthesis to obtain synthesized speech with a target timbre, thereby enabling voice interaction with the user. The server can be a standalone server, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms; it can also be a service node in a blockchain system, where the service nodes form a peer-to-peer (P2P) network. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP). A demand management system server can be installed on the server, allowing interaction with the terminal. For example, corresponding software can be installed on the server, which may implement speech synthesis methods, model training methods, etc., but is not limited to these forms. The terminal and the server can be connected via Bluetooth, USB (Universal Serial Bus), or network communication methods, and this embodiment does not impose any restrictions.

[0085] This invention can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This invention can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This invention can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0086] In one embodiment, the speech synthesis model employs an Encoder-Decoder structure, a deep learning model framework primarily used for sequence-to-sequence learning tasks. In related technologies, speech synthesis models mainly include an Encoder unit and a Decoder unit. The Encoder unit uses a network model to convert the input text sequence into a fixed-length vector, while the Decoder unit uses another network model to convert the vector generated by the Encoder unit into an output vector. Since the Encoder and Decoder units are linked by this fixed vector, during the encoding process, the Encoder unit compresses all the information from the text sequence into this fixed vector, which limits the length of the input text sequence. When the input sequence is short, the model performs well; however, when the input sequence is long, the model's performance deteriorates because the information carried by the earlier input content is diluted by the information carried by the later input content. The longer the input sequence, the more severe this phenomenon becomes. This results in insufficient information from the input sequence being obtained from the beginning during decoding, leading to low decoding accuracy.

[0087] Therefore, this embodiment introduces an attention unit into the speech synthesis model. When generating output, the encoding unit also generates an "attention range," which indicates which parts of the input text sequence should be prioritized for subsequent outputs. The next output is then generated based on the region of focus within the attention range, and this process is repeated. At this point, the encoding unit needs to encode the input text sequence into a sequence of state vectors, rather than a fixed-length context vector. During decoding, the decoding unit utilizes the attention unit to selectively assign different attention weights to the vectors in the state vector sequence at each step, obtaining a context vector for further processing. In this way, the information carried by the input text sequence can be fully utilized when generating each output vector.

[0088] Figure 1 This is an optional flowchart of the model training method provided in the embodiments of the present invention. Figure 1 The method may include, but is not limited to, steps S110 to S150.

[0089] Step S110: Obtain the training dataset.

[0090] In this embodiment, the training dataset includes text sequence samples and their corresponding speech tags. The text sequence samples are the text used for speech synthesis; they can be an article, a section of text, a line of text, or several phrases from an article. The article can cover various fields, such as science and technology, sports, leisure and entertainment, food, and literature. The corresponding speech tags are the pronunciations of the text within the text sequence.

[0091] In actual training, it is usually necessary to obtain a large number of text sequence samples and corresponding speech labels. For the convenience of explanation, one training sample pair will be used as an example for introduction below. For example, a training sample pair includes two parts, namely a text sequence and the corresponding audio. The text sequence is represented as linguistic features. Taking the original text "training model" as an example, its corresponding text sequence is represented as "xun4 lian4 mo2 xing2". Among them, "xun" represents the pronunciation of the character "训", "4" represents the fourth tone of the character "训", "lian" represents the pronunciation of the character "练", "4" represents the fourth tone of the character "练", "mo" represents the pronunciation of the character "模", "2" represents the second tone of the character "模", "xing" represents the pronunciation of the character "型", and "2" represents the second tone of the character "型". The corresponding audio refers to an audio of reading out the text. For example, object A reads the four characters "training model" and records it to obtain a speech signal. The process of processing it includes but is not limited to: for example, if the high-frequency part signal in the audio is weak, pre-emphasis needs to be performed to increase the high-frequency signal, so as to balance the high and low-frequency signals, which can avoid numerical calculation problems in the Fourier transform. Or increase the signal-to-noise ratio. Or after filtering the audio, perform a sliding window Fourier transform on the signal in the time domain. Before the Fourier transform, in order to prevent energy leakage, a window function is used for filtering. After being processed by the short-time Fourier transform, the linear spectrum of the audio is obtained. Generally, the dimension of the linear spectrum is usually high. Therefore, all spectra are taken and all spectra are divided into equally spaced frequencies. Here, the equally spaced frequencies are selected according to the distance that the human ear can hear. Finally, a real Mel spectrum is generated. The method of annotating the text sequence sample can be machine annotation or manual annotation, which is not limited here.

[0092] Step S120: Extract features from the text sequence sample through an encoding unit to obtain a hidden feature vector.

[0093] In this embodiment, the text sequence is input into the encoding unit of the speech synthesis model for feature extraction to generate a hidden feature vector.

[0094] In this embodiment, the text sequence is represented as The hidden feature vector is represented as The encoding unit is the encoding unit. The encoding unit is used to convert text information into high-dimensional text features related to time series, and is used to encode the input text sequence {x} = {x1, x2,..., x L} into a hidden feature vector {h} = {h1, h2,..., h L} to represent the text information.

[0095] In one possible implementation, the encoding unit is an autoregressive recurrent neural network. The encoding unit first converts the input sequence into character vectors, then models the context of the input text sequence through three convolutional layers. Each convolutional layer contains 512 5*1 convolutional kernels. This is followed by batch normalization and activation functions. Finally, the output of the convolutional layers is input into a bidirectional LSTM layer to generate encoded features. This embodiment does not limit the specific network structure of the encoding unit.

[0096] Step S130: Select attention for the latent feature vector through the attention unit to obtain the attention weight sequence.

[0097] In one embodiment, the speech synthesis model uses an attention weight sequence to establish an alignment relationship between the text sequence and the Mel spectrum sequence, and the attention weight sequence is generated by the attention unit.

[0098] Step S140: The latent feature vector and attention weight sequence are decoded by the decoding unit to obtain the speech output vector.

[0099] In one embodiment, reference is made to Figure 2 Step S140 includes, but is not limited to, steps S141 to S143.

[0100] Step S141: Generate a context vector based on the latent feature vector and the attention weight sequence.

[0101] In one embodiment, the attention weight sequence is represented as: The context vector is represented as:

[0102] An attention unit is constructed to accept the output of the encoding unit. For each output of the encoding unit, the attention unit summarizes the encoded latent feature vector into a context vector. Then, the attention weights are used to calculate the context vector at each decoding unit step.

[0103] In this embodiment, the attention weight sequence is obtained by adjusting the attention unit according to the monotonic alignment loss function.

[0104] Step S142: Obtain the state vector generated by the decoding unit of the speech synthesis model.

[0105] In one embodiment, the state vector is represented as

[0106] Step S143: Generate a speech output vector based on the state vector and the context vector.

[0107] In one embodiment, reference is made to Figure 3Step S143 generates a speech output vector based on the state vector and the context vector, and may also include the following steps S1431 to S1432.

[0108] Step S1431: Generate Mel speech vectors based on the state vector and context vector.

[0109] Step S1432: Generate a speech output vector based on the Mel speech vector.

[0110] In this embodiment, the Mel speech vector is represented as: The speech output vector is represented as:

[0111] In one embodiment, the decoding unit first obtains the Mel speech vector using the context vector and the state vector, and then filters the Mel speech vector using a filtering function to obtain the corresponding speech output vector. The Mel speech vector is a speech vector containing Mel spectrum information.

[0112] In one embodiment, the decoding unit predicts the output Mel speech vector, i.e., the frame sequence of Mel spectrograms, based on the context vector. First, the predicted spectrogram frames from the previous step are input into a two-layer fully connected "Pre-Net," with each layer containing 256 hidden ReLU units to learn attention and undergoing regularization after convolutional layers. Then, the context vector and the Pre-Net output are passed through two unidirectional LSTM layers with 1024 units each, followed by two projection layers to predict the completion probability, indicating whether the generated sequence has reached the last frame. During training, if this probability exceeds 0.5, spectrogram frame generation automatically stops. Finally, to improve generation accuracy, a five-convolutional-layer "Post-Net" is introduced as a post-processing network, with each layer consisting of 512 5*1 convolutional kernels and a batch normalization process. The Post-Net refines the Mel spectrograms predicted by the decoding unit using bidirectional context information, and the output of the last layer is superimposed on the spectrogram frames before convolution to produce the final result.

[0113] In this embodiment, the decoding unit is based on the context vector c of the current step. i and the decoding state vector s from the previous step i and the already output speech vector y i Complete the update of the decoding status.

[0114] Step S150: Train the speech synthesis model based on the preset loss function and the monotonic alignment loss function.

[0115] In this embodiment, the speech output vector and speech labels are trained using a preset loss function, the model parameters of the speech synthesis model are adjusted, and the attention weight sequence is trained monotonically based on a monotonic alignment loss function until the value of the loss function converges, thus obtaining a trained speech synthesis model.

[0116] In one embodiment, determining the loss function for training the speech synthesis model specifically involves obtaining a monotonic alignment loss function for training the attention unit and a preset loss function for training the speech synthesis model. Specifically, the loss function includes both a monotonic alignment loss function and a preset loss function. The monotonic alignment loss function characterizes the monotonicity of the attention weight sequence generated by the speech synthesis model, while the preset loss function characterizes the difference between the speech output sequence and the speech labels of the speech synthesis model. In this embodiment, the monotonic alignment loss function is first used to adjust the monotonicity of the attention weight sequence during speech synthesis to ensure that the attention weight sequence is monotonic. Then, the preset loss function is used to adjust the difference between the speech output sequence and the speech labels, thereby training the speech synthesis model.

[0117] In the above embodiments, the speech synthesis model uses attention weight sequences to establish alignment between the text sequence and the Mel spectrum sequence. Since alignment affects the quality of the synthesized speech output, if the attention weights are not precisely aligned, the synthesized speech may be unclear or dissimilar to a human voice. This embodiment ensures the monotonicity of the attention weight sequences by setting a monotonic alignment loss function for attention alignment, enabling precise alignment. This helps the model converge quickly, improves training accuracy, avoids word repetition or omission caused by inaccurate speech synthesis results, and enhances the naturalness and robustness of the synthesized language.

[0118] In one embodiment, the attention unit performs attention selection on the latent feature vectors of each frame of speech in the speech output vector, obtaining an attention weight sequence for each frame of speech in the speech output vector. (Refer to...) Figure 4 In step S150, the attention weight sequence is trained monotonically based on the monotonic alignment loss function, including steps S1501 to S1505.

[0119] Step S1501: Calculate the attention weights of the speech in the current frame based on the attention weight sequence of the speech in the current frame.

[0120] Step S1502: Calculate the attention weight of the speech in the next frame based on the attention weight sequence of the speech in the next frame.

[0121] Step S1503: Calculate the difference between the attention weight of the speech in the current frame and the attention weight of the speech in the next frame to obtain the attention weight difference.

[0122] Step S1504: Calculate the function value of the monotonic alignment loss function based on the attention weight difference.

[0123] In one embodiment, the attention weight difference is input into a preset activation function for calculation to obtain the activation value, wherein the preset activation function may be a ReLU activation function. Then, the activation values ​​of all intra-frame speech in the speech output vector are accumulated to obtain the function value of the monotonic alignment loss function.

[0124] Step S1505: Adjust the attention weight sequence according to the function value until the value of the monotonic alignment loss function meets the preset condition.

[0125] In one embodiment, the monotonic alignment loss function is expressed as:

[0126]

[0127] Relu(x) = max(x,0)

[0128] Where Loss1 represents the monotonic alignment loss function, T represents the total number of frames in the speech output vector, and a i This represents the attention weight sequence of the i-th frame of speech. Let a represent the attention weight of the i-th frame of speech. i+1 This represents the attention weight sequence of the (i+1)th frame of speech. This represents the attention weight of the speech in the (i+1)th frame.

[0129] In this embodiment, reference is made to Figure 5 The steps for calculating attention weights include steps S1506 to S1508.

[0130] Step S1506: Calculate the target position of the attention weight sequence of the speech in the current frame.

[0131] In one embodiment, take a i The index j corresponding to the maximum value in the middle max Let j be the target position. Suppose that the j-th text in the text sequence corresponds to the i-th frame of the speech, then the i-th frame of the speech is the center position of the text. At this center position, the probability that text j corresponds to the i-th frame of the speech is the highest.

[0132] Step S1507: Calculate the probability that the speech in the current frame is the text at the target position in the text sequence based on the target position.

[0133] In one embodiment, the i-th frame of speech belongs to the j-th frame. max When there is a text, The value is the largest. That is... This represents the probability that the speech corresponding to the current speech frame belongs to the target text in the text sequence.

[0134] Step S1508: Calculate the attention weight corresponding to the current speech frame number based on the target location and probability.

[0135] In one embodiment, the attention weights are represented as:

[0136]

[0137]

[0138] Where L represents the number of texts in the text sequence, a i,j Let a represent the probability that the speech in the i-th frame is the text at the j-th position in the text sequence. In a two-dimensional plane, a i,j The probability value increases as the distance from the diagonal increases, and decreases as the distance increases.

[0139] In this embodiment, the monotonic alignment loss function is used to adjust the monotonicity of the attention weight sequence during speech synthesis, ensuring that the attention weight sequence is monotonic. The convergence condition of the monotonic alignment loss function is set as follows: the function value of the monotonic alignment loss function tends to be negative. Because the text is pronounced sequentially, the alignment relationship between the text position j and the speech frame number i should be strictly monotonically increasing. Strictly increasing means that if the speech frame number i corresponds to the text position j, then the speech frame number i+1 corresponds to the text position j or j+1, ensuring... If the (i+1)th frame of speech belongs to the nth text, then n = j must be satisfied. max +1 or n = j max The monotonic alignment loss function optimizes what? The value tends towards negative to satisfy the monotonicity of the alignment relationship. It requires no additional network structure or computational cost, is applicable to different attention units, does not increase model training costs, and has universality. Experimental results show that using this monotonic alignment loss function can effectively save model training time and improve convergence stability. It also significantly improves the naturalness and robustness of synthesized speech.

[0140] In the above embodiment, the softmax function is used to ensure the monotonicity of the alignment relationship.

[0141] Reference Figure 6 This diagram illustrates the attention mechanism of the speech synthesis model in this embodiment. As can be seen from the diagram, the input text sequence enters the encoding unit to obtain a state vector, which is then combined with the corresponding context vector representing the attention range and enters the decoding unit. The decoding unit generates a speech output vector, which is obtained based on the context vector representing the attention.

[0142] The input text sequence is represented as follows: L represents the number of texts in the input text sequence, and the context vector is represented as: The speech output vector is represented as: T represents the number of speech frames in the speech output vector; the number of text frames differs from the number of speech frames. In one embodiment, the calculation process of step i during training is described as follows:

[0143] h = Encode(x)

[0144]

[0145] s i =Decode(s i-1 ,[c i-1 ,y i-1 ])

[0146] Among them, c i Let a represent the context vector. i s represents the sequence of attention weights generated by the attention unit. i This represents the state vector, Encode represents the encoding unit, and Decode represents the decoding unit.

[0147] In the above embodiments, the attention unit's role is to calculate the elements in the latent feature vector that the decoding unit focuses on. For these key elements, a weighted average is applied, effectively improving the model's modeling accuracy. During alignment, the attention unit generates the attention weight sequence a. i The weights are primarily represented by the state vector from the previous step of the decoding unit. If it's a content-based attention unit, the state s from the previous step of the decoding unit is used when generating the attention weight sequence. i-1 And the calculation of latent feature vectors; in this case, content-based attention units are generally implemented using multilayer perceptrons. If it is a position-based attention unit, then the generation of the attention weight sequence mainly depends on the state s of the decoding unit in the previous step. i-1 And the attention weight sequence value a obtained in the previous step i-1 Calculation. In this embodiment, the attention unit can be content-based, location-based, or a combination of both; no specific limitation is made here.

[0148] In one embodiment, for each speech output vector, the information provided at different times in the input text sequence has different effects. The attention unit selects or weights the input frames (i.e., the latent feature vectors output by the encoding unit), giving high weight to important information and low weight to less important information, thus fully utilizing more effective information to generate the current output and more accurately capturing useful information to predict the output. In this embodiment, the attention unit mainly performs the task of adjusting the attention weight sequence a. i The calculation, and based on ai Calculate attention weight a i,j Then through a i,j Calculate the context vector. Where the attention weights a... i,j It is a i The j-th element represents the probability that the speech corresponding to speech frame i belongs to the j-th text in the text sequence.

[0149] In one embodiment, the network parameters of the speech synthesis model are updated by calculating the value of a preset loss function. The preset loss function can be a loss function used in related technologies, such as the cross-entropy loss function. The specific form of the preset loss function is not limited here.

[0150] In a specific application scenario, suppose the input text sequence is {x1, x2, ..., x...} L The terminal inputs the text sequence to the encoding unit, which processes the input text sequence {x1, x2, ..., x...}. L Encode the vector to obtain a latent feature vector {h1, h2, ..., h} containing semantic features. L During the decoding process, the attention unit determines the level of attention required for each element in the latent feature vector in real time, and applies different weights to elements with different levels of attention (i.e., selects different values ​​for each element). i,j Thus, the decoding unit can obtain the Mel spectrum features {d1,d2,...,d...}. T Then, the Mel spectrum features {d1,d2,...,d...} are filtered using a filtering function (e.g., inverse Fourier transform). T} is transformed into a speech output vector {y1,y2,...,y} T It should be noted that the decoding unit in this embodiment is an autoregressive structure. The initial state is d0, d1 is generated based on d0, and then d2 is generated based on d1, and so on, to obtain the Mel spectrum features {d1, d2, ..., d...}. T}

[0151] The technical solution provided in this invention involves acquiring a training dataset, determining a loss function for training a speech synthesis model, and constructing a loss function that includes a monotonic alignment loss function to characterize the monotonicity of the attention weight sequence generated by the speech synthesis model and a preset loss function to characterize the difference between the speech output sequence and the speech labels of the speech synthesis model. Finally, the speech synthesis model is trained based on the loss function and the training dataset until the value of the loss function meets a preset condition, resulting in a well-trained speech synthesis model. This embodiment sets a monotonic alignment loss function for attention alignment to ensure the monotonicity of the attention weight sequence, which helps the model converge quickly and improves the accuracy of model training. It also avoids problems such as word repetition or omission caused by inaccurate speech synthesis results, thereby improving the naturalness and robustness of the synthesized language.

[0152] After the above training is completed, refer to Figure 7 This invention also provides a speech synthesis method, comprising:

[0153] Step S710: Obtain the text sequence to be synthesized;

[0154] Step S720: Input the text sequence to be synthesized into the speech synthesis model for speech synthesis processing to obtain the speech output sequence.

[0155] In one embodiment, the speech synthesis model is trained using the model training method described in any of the preceding embodiments.

[0156] In one application scenario, this method is applied to a reading and audiobook software on a terminal, with a pre-trained speech synthesis model built into the software. Assuming the user is in a no-network or weak-network environment and wants to use the reading and audiobook function, the interface displays different reading materials. The user can then select the desired content, and the software inputs this content as a text sequence to the speech synthesis model. The model then performs a series of encoding and decoding steps to convert the reading content into synthesized output, which the user can then hear. In this embodiment, the speech synthesis model uses a monotonic alignment loss function to ensure the monotonicity of the attention weight sequence, improving the accuracy of the synthesis results, avoiding word repetition or omission, enhancing the naturalness and robustness of the synthesized language, and improving the user experience.

[0157] This invention also provides a model training apparatus for training a speech synthesis model. The speech synthesis model includes an encoding unit, an attention unit, and a decoding unit, and can implement the above-described model training method. (Refer to...) Figure 8 The device includes:

[0158] The training data acquisition module 810 is used to acquire the training dataset, which includes text sequence samples and speech labels for the text sequence samples.

[0159] The encoding module 820 is used to extract features from text sequence samples through the encoding unit to obtain latent feature vectors;

[0160] Attention module 830 is used to select attention from the latent feature vector through attention units to obtain the attention weight sequence;

[0161] The decoding module 840 is used to decode the latent feature vector and attention weight sequence through the decoding unit to obtain the speech output vector;

[0162] The training module 850 is used to train the speech output vector and speech labels using a preset loss function, adjust the model parameters of the speech synthesis model, and perform monotonic training on the attention weight sequence based on the monotonic alignment loss function until the value of the loss function converges, thus obtaining the trained speech synthesis model.

[0163] The specific implementation of the speech synthesis model training device in this embodiment is basically the same as the specific implementation of the model training method described above, and will not be repeated here.

[0164] This invention also provides a speech synthesis device, referring to... Figure 9 The device includes:

[0165] Module 910 is used to acquire the text sequence to be synthesized;

[0166] The synthesis module 920 is used to input the text sequence to be synthesized into the speech synthesis model for speech synthesis processing to obtain the speech output sequence. The speech synthesis model is trained using the model training method described above.

[0167] This invention also provides an electronic device, comprising:

[0168] At least one memory;

[0169] At least one processor;

[0170] At least one program;

[0171] The program is stored in a memory, and the processor executes the at least one program to implement the speech synthesis method and model training method described above in this invention. The electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0172] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0173] The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0174] The memory 1002 can be implemented in the form of ROM (Read-Only Memory), static storage device, dynamic storage device, or RAM (Random Access Memory). The memory 1002 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 to execute the speech synthesis method and model training method of the embodiments of this invention.

[0175] Input / output interface 1003 is used to implement information input and output;

[0176] Communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved via wired means (e.g., USB, Ethernet cable) or wireless means (e.g., mobile network, Wi-Fi, Bluetooth).

[0177] Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004);

[0178] The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0179] This invention also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described speech synthesis method and model training method.

[0180] The model training method and apparatus, speech synthesis method, device, and storage medium proposed in this invention acquire a training dataset, then determine a loss function for training the speech synthesis model. The constructed loss function includes a monotonic alignment loss function characterizing the monotonicity of the attention weight sequence generated by the speech synthesis model, and a preset loss function characterizing the difference between the speech output sequence and the speech labels of the speech synthesis model. Finally, the speech synthesis model is trained based on the loss function and the training dataset until the value of the loss function meets a preset condition, resulting in a trained speech synthesis model. Setting a monotonic alignment loss function for attention alignment ensures the monotonicity of the attention weight sequence, which helps the model converge quickly and improves the accuracy of model training. It also avoids problems such as word repetition or omission caused by inaccurate speech synthesis results, thus improving the naturalness and robustness of the synthesized language.

[0181] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0182] The embodiments described in this invention are for the purpose of more clearly illustrating the technical solutions of the embodiments of this invention, and do not constitute a limitation on the technical solutions provided by the embodiments of this invention. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this invention are also applicable to similar technical problems.

[0183] It will be understood by those skilled in the art that Figure 1-10 The technical solutions shown do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0184] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0185] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0186] The terms "first," "second," "third," "fourth," etc. (if present) in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0187] It should be understood that in this invention, "at least one (item)" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0188] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0189] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0191] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0192] The preferred embodiments of the present invention have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present invention should be within the scope of the claims of the present invention.

Claims

1. A model training method for training a speech synthesis model, the speech synthesis model comprising an encoding unit, an attention unit, and a decoding unit, characterized in that, include: Obtain a training dataset, which includes text sequence samples and speech labels for the text sequence samples; The text sequence sample is subjected to feature extraction by the encoding unit to obtain a latent feature vector; The attention unit selects attention for the latent feature vector for each frame of speech in the speech output vector, thereby obtaining the attention weight sequence for each frame of speech in the speech output vector. The latent feature vector and the attention weight sequence are decoded by the decoding unit to obtain the speech output vector; The speech output vector and the speech label are trained by a preset loss function, the model parameters of the speech synthesis model are adjusted, and the attention weight of the speech in the current frame is calculated based on the attention weight sequence of the speech in the current frame. Calculate the attention weights of the speech in the next frame based on the attention weight sequence of the speech in the next frame; Calculate the difference between the attention weight of the speech in the current frame and the attention weight of the speech in the next frame to obtain the attention weight difference; The difference is input into a preset activation function for calculation to obtain the activation value; The activation values ​​of the speech in all frames of the speech output vector are summed to obtain the function value of the monotonic alignment loss function; The attention weight sequence is adjusted according to the function value until the value of the monotonic alignment loss function meets the preset condition and the value of the preset loss function converges, thus obtaining a trained speech synthesis model.

2. The model training method according to claim 1, characterized in that, The step of decoding the latent feature vector and the attention weight sequence through the decoding unit to obtain the speech output vector includes: Generate a context vector based on the latent feature vector and the attention weight sequence; Obtain the state vector generated by the decoding unit of the speech synthesis model; A speech output vector is generated based on the state vector and the context vector.

3. The model training method according to claim 1, characterized in that, The activation function is the ReLU function, and the monotonic alignment loss function is expressed as follows: in, Let T represent the monotonic alignment loss function, and let T represent the total number of frames in the speech output vector. The attention weight sequence represents the i-th frame of speech. The attention weights represent the values ​​in the i-th frame of speech. The attention weight sequence represents the (i+1)th frame of speech. The attention weight represents the speech in the (i+1)th frame.

4. The model training method according to claim 3, characterized in that, The calculation of the attention weights of the speech in the current frame based on the attention weight sequence of the speech in the current frame is expressed as follows: Calculate the target location of the attention weight sequence of the speech in the current frame; Calculate the probability that the speech in the current frame is the text at the target location in the text sequence sample based on the target location; The attention weight of the speech in the current frame is calculated based on the target location and the probability. Represented as: Where L represents the number of texts in the text sequence sample, and j represents the target position. This represents the probability that the speech in the i-th frame is the text at the j-th position in the text sequence sample.

5. The model training method according to claim 2, characterized in that, The step of generating a speech output vector based on the state vector and the context vector includes: Generate Mel speech vectors based on the state vector and the context vector; The speech output vector is generated based on the Mel speech vector.

6. A speech synthesis method, characterized in that, include: Obtain the text sequence to be synthesized; The text sequence to be synthesized is input into the speech synthesis model for speech synthesis processing to obtain a speech output sequence. The speech synthesis model is trained using the model training method described in any one of claims 1 to 5.

7. A model training apparatus for training a speech synthesis model, the speech synthesis model comprising an encoding unit, an attention unit, and a decoding unit, characterized in that, include: The training data acquisition module is used to acquire a training dataset, which includes text sequence samples and speech tags for the text sequence samples. An encoding module is used to extract features from the text sequence sample through the encoding unit to obtain a latent feature vector; The attention module is used to select attention for the latent feature vector for each frame of speech in the speech output vector through the attention unit, so as to obtain the attention weight sequence of speech for each frame of speech in the speech output vector. The decoding module is used to decode the latent feature vector and the attention weight sequence through the decoding unit to obtain the speech output vector; The training module is used to train the speech output vector and the speech label using a preset loss function, adjust the model parameters of the speech synthesis model, and calculate the attention weight of the speech in the current frame based on the attention weight sequence of the speech in the current frame. Calculate the attention weights of the speech in the next frame based on the attention weight sequence of the speech in the next frame; Calculate the difference between the attention weight of the speech in the current frame and the attention weight of the speech in the next frame to obtain the attention weight difference; The difference is input into a preset activation function for calculation to obtain the activation value; The activation values ​​of the speech in all frames of the speech output vector are summed to obtain the function value of the monotonic alignment loss function; The attention weight sequence is adjusted according to the function value until the value of the monotonic alignment loss function meets the preset condition and the value of the preset loss function converges, thus obtaining a trained speech synthesis model.

8. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor is configured to execute, according to the program, the model training method as described in any one of claims 1 to 5, or the speech synthesis method as described in claim 6.

9. A computer-readable storage medium storing computer-executable instructions for performing the model training method of any one of claims 1 to 5, or the speech synthesis method of claim 6.

Images