Speech synthesis model training method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a speech synthesis model training method and device, equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a phoneme sequence of a text sample and real acoustic characteristics of voice of the text sample; inputting the phoneme sequence and the real acoustic features into an improved TTS acoustic model for training; through an OT-CFM mechanism, an ordinary differential equation is guided to perform acoustic feature generation processing on the phoneme sequence, and predicted acoustic features aligned with expected acoustic features of the text sample are obtained; and according to the aligned predicted acoustic features and real acoustic features, parameter updating is performed on the improved TTS acoustic model to obtain a trained speech synthesis model, and the trained speech synthesis model is used for synthesizing the speech of the text. According to the embodiment of the invention, the training difficulty and the training cost can be reduced, and a speech synthesis model with a higher synthesis speed and a higher synthesis effect is provided, so that the quality of synthesized speech is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, device and storage medium for a speech synthesis model. Background Art

[0002] With the development of deep learning technology, deep learning can be applied to the task of text-to-speech. Using diffusion probabilistic models (DPMs) to synthesize speech from text is one of the research directions.

[0003] However, the diffusion probability model relies on multiple reverse steps for gradual denoising, and each denoising step takes a certain amount of time, resulting in a slow speech synthesis speed. In addition, the diffusion probability model is a process of restoring the input signal through gradual denoising, and this process may produce a certain amount of ambiguity or uncertainty. Some details (such as the clarity of pronunciation, the accuracy of pitch, etc.) may still appear to be relatively vague or unstable, resulting in the synthesized speech lacking in naturalness and clarity, affecting the quality of the synthesized speech. In addition, due to the complexity of the diffusion probability model and the fact that the speech synthesis process requires multiple iterations and the processing of complex probability distributions, the training is difficult and requires a large amount of computing resources, increasing the training cost.

[0004] Therefore, how to provide a speech synthesis model with faster synthesis speed and higher synthesis effect while reducing the training difficulty and training cost to improve the quality of synthesized speech has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to propose a training method, device, equipment and storage medium for a speech synthesis model, aiming to reduce the difficulty and cost of training, provide a speech synthesis model with faster synthesis speed and higher synthesis effect, so as to improve the quality of synthesized speech.

[0006] To achieve the above object, a first aspect of an embodiment of the present application proposes a method for training a speech synthesis model, the method comprising:

[0007] By obtaining a phoneme sequence of a text sample and a real acoustic feature of the speech of the text sample;

[0008] Inputting the phoneme sequence and the real acoustic features into an improved TTS acoustic model for training;

[0009] By using the OT-CFM mechanism, an ordinary differential equation is guided to perform acoustic feature generation processing on the phoneme sequence to obtain a predicted acoustic feature aligned with the expected acoustic feature of the text sample;

[0010] According to the aligned predicted acoustic features and the real acoustic features, the parameters of the improved TTS acoustic model are updated until the improved TTS acoustic model converges to obtain a trained speech synthesis model, and the trained speech synthesis model is used to synthesize the speech of the text.

[0011] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a training device for a speech synthesis model, the device comprising:

[0012] A data acquisition module, used to acquire a phoneme sequence of a text sample and real acoustic features of the speech of the text sample;

[0013] A model training module, used for inputting the phoneme sequence and the real acoustic features into the TTS acoustic model for training;

[0014] A feature alignment module, used to guide the ordinary differential equation to perform acoustic feature generation processing on the phoneme sequence through the OT-CFM mechanism, so as to obtain a predicted acoustic feature aligned with the expected acoustic feature of the text sample;

[0015] A parameter updating module is used to update the parameters of the improved TTS acoustic model according to the predicted acoustic features and the real acoustic features until the improved TTS acoustic model converges to obtain a trained speech synthesis model, and the trained speech synthesis model is used to synthesize the speech of the text.

[0016] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes a computer device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0017] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0018] The training method, apparatus, device and storage medium of the speech synthesis model proposed in the present application, the training method of the speech synthesis model, the training method of the speech synthesis model provided in the above-mentioned embodiments, obtain the phoneme sequence of the text sample and the real acoustic features of the speech of the text sample; input the phoneme sequence and the real acoustic features into the improved TTS acoustic model for training; guide the ordinary differential equation to perform acoustic feature generation processing on the phoneme sequence through the optimal transport condition flow matching OT-CFM mechanism, and obtain the predicted acoustic features aligned with the expected acoustic features of the text sample; according to the aligned predicted acoustic features and the real acoustic features, update the parameters of the improved TTS acoustic model until the improved TTS acoustic model converges to obtain the trained speech synthesis model, and the trained speech synthesis model is used to synthesize the speech of the text. In this way, by guiding the ordinary differential equation to generate predicted acoustic features that are aligned with the expected acoustic features through the OT-CFM mechanism, predicted acoustic features that are closer to the actual acoustic features can be obtained within fewer generation steps, so as to accelerate the convergence process of the improved TTS acoustic model based on the predicted acoustic features that are closer to the actual acoustic features, so that the improved TTS acoustic model can achieve better speech synthesis quality in fewer training iterations, thereby reducing the training difficulty and training cost, and providing a speech synthesis model with faster synthesis speed and higher synthesis effect, so as to improve the quality of synthesized speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of a method for training a speech synthesis model provided in an embodiment of the present application;

[0020] Figure 2 Schematic diagram of the training architecture for training an improved TTS acoustic model provided in an embodiment of the present application;

[0021] Figure 3 is another flow chart of the training method of the speech synthesis model provided in an embodiment of the present application;

[0022] Figure 4 is a schematic block diagram of a training device for a speech synthesis model provided in an embodiment of the present application;

[0023] Figure 5 It is a schematic block diagram of the structure of the computer device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0025] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0027] First, some nouns involved in this application are analyzed:

[0028] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0029] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0030] Text-to-Speech (TTS) technology: refers to the technology that converts written text into natural speech.

[0031] Improved TTS acoustic model: refers to the improved acoustic model based on TTS. The improved TTS acoustic model is a new TTS architecture that combines the Optimal Transport-Conditional Flow Matching (OT-CFM) mechanism and the hybrid convolution-Transformer decoder based on the ordinary differential equation (ODE). Under the conditions of non-autoregression and fast sampling, the improved TTS acoustic model can show superior speed and high synthesis quality in synthesizing speech of long texts.

[0032] Among them, the improved TTS acoustic model mainly includes the encoder-decoder architecture.

[0033] It is worth noting that the decoder is an improved U-Net structure, which mainly includes a convolutional neural network (CNN) and a Transformer, where the convolutional neural network can be a 1D convolutional neural network. This structure of the decoder can simultaneously optimize the memory consumption and computing speed of the improved TTS acoustic model during the training process, so that the improved TTS acoustic model can learn to generate acoustic features (also called audio features, such as Mel-spectrograms) more efficiently during the training process, ensuring a fast speech synthesis process and improving the training speed and training effect.

[0034] Specifically, the 1D convolutional neural network is used to capture the local context of the phoneme sequence, while the Transformer is used to capture the long-distance dependencies and can capture the global temporal information of the phoneme sequence.

[0035] It is worth mentioning that the decoder uses ordinary differential equations (ODE) to generate acoustic features.

[0036] Ordinary differential equation: is a mathematical tool used to describe the relationship between one or more unknown functions and their derivatives and independent variables. The characteristic of ordinary differential equations is that they involve only one independent variable, that is, the relationship between the unknown function and its derivative depends on only one independent variable.

[0037] In the decoder, the core role of ordinary differential equations is to control how acoustic features change over time. Through ordinary differential equations, the decoder can smoothly generate acoustic features in the time dimension, thus ensuring that the generation process has smooth continuity.

[0038] Among them, Transformer does not use position embedding, because the position information in the text can be embedded into the improved TTS acoustic model through the encoder. And the snake beta activation function is used in the feedforward network of Transformer, which helps to capture more complex nonlinear relationships.

[0039] In this way, the encoder that integrates 1D convolutional neural networks and Transformer can process local features and long-range dependencies at the same time, so that the improved TTS acoustic model can effectively understand the temporal relationship of phonemes, ensuring the efficient performance of the improved TTS acoustic model on long texts; and the encoder uses ordinary differential equations to make the decoding process not just a mapping between discrete time steps, but takes into account smoother time evolution and continuity, which complements the feature flow in 1D convolutional neural networks and Transformer, so that the improved TTS acoustic model can effectively learn to synthesize higher quality and more natural speech while reducing the synthesis steps during training.

[0040] It is worth noting that the non-autoregressive architecture means that the improved TTS acoustic model can generate output in parallel without relying on previous outputs, so that the improved TTS acoustic model can directly learn how to map from text input to speech output, which makes the speech synthesis process more parallel and can significantly improve the speed of speech synthesis, especially when processing long texts.

[0041] Optimal Transport (OT): is a mathematical method used to find an optimal matching mapping between different distributions to minimize the "transmission cost" between the two.

[0042] Conditional Flow Matching (CFM): is a technique in generative models that aims to train the model by matching the flow characteristics of sample distributions.

[0043] OT-CFM mechanism: aims to simplify the training process and improve the quality of speech synthesis. It is a flow matching based on optimal transport theory, which improves the traditional continuous normalized flows (CNFs) and score matching probability flows (SMPF). It defines a simplified path from a source distribution to a target distribution, so that the improved TTS acoustic model can effectively learn to predict high-quality acoustic features in fewer steps (compared to the diffusion probability model DPMs).

[0044] Based on this, the embodiments of the present application provide a training method, apparatus, device and storage medium for a speech synthesis model, aiming to reduce the difficulty and cost of training, and provide a speech synthesis model with faster synthesis speed and higher synthesis effect, so as to improve the quality of synthesized speech.

[0045] The training method, apparatus, device and storage medium of the speech synthesis model provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the training method of the speech synthesis model in the embodiments of the present application is described.

[0046] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0047] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0048] The training method of the speech synthesis model provided in the embodiment of the present application relates to the field of artificial intelligence technology. The training method of the speech synthesis model provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a vehicle-mounted terminal, a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application of the training method of the speech synthesis model, etc., but is not limited to the above forms.

[0049] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer computer devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0050] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0051] Figure 1 is an optional flowchart of the method for training a speech synthesis model provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S104.

[0052] Step S101, obtaining a phoneme sequence of a text sample and real acoustic features of the speech of the text sample.

[0053] Get text samples for training an improved TTS acoustic model.

[0054] Exemplarily, the text sample may be a financial text sample, such as financial reports, financial product introductions and other text data used in the financial technology field; the text sample may also be a medical text sample, such as physical examination reports, clinical notes and other text data used in the medical field; the text sample may also be other news material type text and other text data.

[0055] In order to improve the understanding of the TTS acoustic model, the text sample can be converted to phonemes to obtain the phoneme sequence of the text sample. Phoneme refers to the smallest unit of speech that can distinguish the meaning of a word. It represents the smallest sound in speech that can distinguish different words or word forms. The phoneme sequence of the text sample provides the standard pronunciation of the text sample.

[0056] The real speech of the text sample may also be processed for acoustic feature extraction to obtain the real acoustic features of the speech of the text sample.

[0057] It should be noted that the real acoustic features are used to calculate the prediction loss of the improved TTS acoustic model as a whole together with the aligned predicted acoustic features.

[0058] Step S102: input the phoneme sequence and the real acoustic features into the improved TTS acoustic model for training.

[0059] See also Figure 2 , Figure 2 The training architecture diagram for training and improving the TTS acoustic model. Figure 2 The improved TTS acoustic model architecture is shown in FIG. 1 , which includes an encoder, a duration predictor, and a decoder. Figure 2 , the specific process of training and improving the TTS acoustic model is explained in detail.

[0060] The phoneme sequence and the real acoustic features of the text sample are used as the input of the improved TTS acoustic model, and the improved TTS acoustic model is effectively trained.

[0061] Step S103, through the OT-CFM mechanism, guide the ordinary differential equation to perform acoustic feature generation processing on the phoneme sequence to obtain the predicted acoustic feature aligned with the expected acoustic feature.

[0062] During the training process, the OT-CFM mechanism is used to guide the ordinary differential equation to generate acoustic features for the phoneme sequence, and obtain predicted acoustic features that are aligned with the expected acoustic features of the text sample. This can simplify the training process and generate predicted acoustic features that are closer to the actual acoustic features in fewer synthesis steps, reducing memory consumption and speeding up the synthesis speed.

[0063] In step S103 of some embodiments, the decoder of the TTS acoustic model can be improved and acoustic feature generation processing can be performed on the phoneme sequence according to ordinary differential equations to obtain predicted acoustic features of the text sample; the acoustic feature calculation processing can be performed on the phoneme sequence through the OT-CFM mechanism to obtain the expected acoustic features of the text sample, and the predicted acoustic features can be aligned with the expected acoustic features to obtain aligned predicted acoustic features.

[0064] In some embodiments, by improving the decoder of the TTS acoustic model, performing acoustic feature generation processing on the phoneme sequence according to ordinary differential equations to obtain the predicted acoustic features of the text sample, the encoder of the TTS acoustic model can be improved to obtain the hidden state vector of the phoneme sequence; by improving the duration predictor of the TTS acoustic model, the duration of the phoneme sequence is predicted according to the hidden state vector; the hidden state vector is sampled according to the duration to obtain the predicted average acoustic features of the text sample, wherein the predicted average acoustic features are used to condition the decoder.

[0065] After the phoneme sequence, real acoustic features and expected acoustic features of the text sample are input into the improved TTS acoustic model, the phoneme sequence of the text sample is first encoded in the encoder of the improved TTS acoustic model to obtain the encoding vector of the phoneme sequence output by the encoder; the encoding vector of the phoneme sequence is then mapped through the mapping layer to obtain the hidden state vector of the phoneme sequence output by the encoder, and the duration of each phoneme in the phoneme sequence is predicted according to the hidden state vector of the phoneme sequence through the duration predictor to obtain the predicted duration. Then, the hidden state vector output by the encoder is quickly upsampled by the predicted duration to obtain the predicted average acoustic feature μ of the text sample at the predicted duration. The predicted average acoustic feature is input into the decoder for conditioning the decoder so that the subsequent decoder can efficiently generate the predicted acoustic feature υ of the text sample. t (φ OT (x0)μ;θ), where θ is the parameter for improving the TTS acoustic model.

[0066] After conditioning the decoder with the predicted average acoustic features, the decoder performs acoustic feature generation processing on the phoneme sequence according to ordinary differential equations to more accurately describe the evolution of the acoustic features in the time dimension and obtain the predicted acoustic features of the text sample.

[0067] Specifically, during the training process, the decoder generates the predicted acoustic features smoothly in the time dimension through ordinary differential equations. The ordinary differential equations are as follows:

[0068]

[0069] Among them, φ t (x) represents the predicted acoustic features of the decoder at time step t;

[0070] υ t represents the velocity field of the decoder at time step t, controlling the change of the predicted acoustic features.

[0071] Thus, in the decoder, the ordinary differential equations define the continuous evolution from the phoneme sequence to the predicted acoustic features. In other words, the ordinary differential equations describe the time flow and transformation from the phoneme sequence to the predicted acoustic features.

[0072] In this way, the temporal relationship between the aligned phoneme sequence and the predicted acoustic features is achieved, which can more accurately describe the evolution of the predicted acoustic features in the time dimension, thereby ensuring that the generation process of the predicted acoustic features has smooth continuity.

[0073] During the training process, the expected acoustic features of the text samples are also calculated through the OT in the OT-CFM mechanism. The expected acoustic features are the ideal acoustic features of the text samples, that is, the acoustic features generated by the expected improved TTS acoustic model during the training process, which are very close to the real acoustic features.

[0074] It should be noted that the expected acoustic features are used to guide the training process so as to improve the TTS acoustic model to generate predicted acoustic features for text samples during the training process that are closer to the expected acoustic features, that is, to ultimately generate predicted acoustic features that are aligned with the expected acoustic features.

[0075] See also Figure 3 In some embodiments, the predicted acoustic features are aligned with the expected acoustic features to obtain aligned predicted acoustic features, which may include but is not limited to steps S301 to S302.

[0076] Step S301, calculating the OT-CFM loss function according to the predicted acoustic features and the target acoustic features;

[0077] Step S302: align the predicted acoustic features with the expected acoustic features according to the OT-CFM loss function to obtain aligned predicted acoustic features.

[0078] In improving the TTS acoustic model, OT can align the predicted acoustic features generated by the decoder with the expected acoustic features with the least "cost".

[0079] CFM is a process of matching features in time. In this process, the decoder continuously adjusts the generated predicted acoustic features to align the predicted acoustic features with the expected acoustic features as much as possible. OT-CFM can make the generated predicted acoustic features have a better alignment effect on the time axis.

[0080] That is, the goal of the OT-CFM mechanism is to minimize the difference between the predicted acoustic features and the expected acoustic features. This is usually achieved by defining an OT-CFM loss function that quantifies the difference between the predicted acoustic features and the expected acoustic features.

[0081] Specifically, in order to further optimize the alignment process between the predicted acoustic features and the expected acoustic features, the OT-CFM mechanism introduces the concept of OT and constructs the OT-CFM loss function. The OT-CFM loss function is as follows:

[0082]

[0083] Among them, q(x1) represents the distribution of phoneme sequence x1;

[0084] P0(x0) represents the distribution of the initial predicted acoustic features x0;

[0085] represents the initial predicted acoustic feature x0 through the dynamic evolution of ordinary differential equations;

[0086] represents the expected acoustic feature calculated by OT, which is obtained by time evolution of the initial predicted acoustic feature x0 based on the phoneme sequence x1 through ordinary differential equations;

[0087] υ t (φ OT (x0)μ;θ) represents the predicted acoustic features generated by the decoder.

[0088] In this way, the optimal transport condition flow matching (OT-CFM) mechanism optimizes the decoder to generate predicted acoustic features that are aligned with the expected acoustic features more smoothly and accurately by combining ordinary differential equations and OT.

[0089] In step S302 in some embodiments, the OT-CFM loss function may be minimized to obtain aligned predicted acoustic features.

[0090] By minimizing the OT-CFM loss function, the difference between the predicted acoustic features and the expected acoustic features can be minimized, ensuring that the generation process of the predicted acoustic features is smoother in time series and accurately aligns with the expected acoustic features, ultimately obtaining predicted acoustic features that are aligned with the expected acoustic features.

[0091] Step S104, updating parameters of the improved TTS acoustic model according to the aligned predicted acoustic features and the real acoustic features until the improved TTS acoustic model converges to obtain a trained speech synthesis model, and the trained speech synthesis model is used to synthesize the speech of the text.

[0092] In this way, the parameters of the improved TTS acoustic model can be updated according to the aligned predicted acoustic features and the actual acoustic features until the improved TTS acoustic model converges to obtain a trained speech synthesis model.

[0093] Therefore, the training process is simplified through the OT-CFM mechanism, and the innovation of the decoder architecture reduces memory consumption and speeds up the training speed, thereby providing a speech synthesis model with faster synthesis speed and higher synthesis effect while reducing the training difficulty and training cost.

[0094] In step S104 of some embodiments, a loss function between the aligned predicted acoustic features and the actual acoustic features can be calculated; based on the loss function, the parameters of the improved TTS acoustic model are iteratively updated until the number of iterations is greater than a preset threshold, and it is determined that the improved TTS acoustic model converges to obtain a trained speech synthesis model.

[0095] Exemplarily, the mean square error (MSE) between the aligned predicted acoustic features and the true acoustic features can be calculated as the loss function. By minimizing the loss function, the parameters of the improved TTS acoustic model are iteratively updated until the number of iterations is greater than a preset threshold, and it is determined that the improved TTS acoustic model converges to obtain a trained speech synthesis model.

[0096] In this way, the loss function can better guide the improved TTS acoustic model to update its parameters, thereby improving the training effect of the improved TTS acoustic model.

[0097] The trained speech synthesis model has fast synthesis capabilities and low memory requirements when synthesizing speech from text. It can synthesize natural and high-quality speech in a short time. It is suitable for application scenarios in the fields of financial technology, medical health, etc. that require real-time speech synthesis (such as intelligent assistants for financial product recommendations in the financial technology field, voice navigation for medical treatment in medical technology, etc.).

[0098] The training method of the speech synthesis model provided in the above embodiment obtains the phoneme sequence of the text sample and the real acoustic features of the speech of the text sample; inputs the phoneme sequence and the real acoustic features into the improved TTS acoustic model for training; guides the ordinary differential equation to perform acoustic feature generation processing on the phoneme sequence through the optimal transport condition flow matching OT-CFM mechanism to obtain the predicted acoustic features aligned with the expected acoustic features of the text sample; updates the parameters of the improved TTS acoustic model according to the aligned predicted acoustic features and the real acoustic features until the improved TTS acoustic model converges to obtain the trained speech synthesis model, and the trained speech synthesis model is used to synthesize the speech of the text. In this way, by guiding the ordinary differential equation to generate predicted acoustic features that are aligned with the expected acoustic features through the OT-CFM mechanism, predicted acoustic features that are closer to the actual acoustic features can be obtained within fewer generation steps, so as to accelerate the convergence process of the improved TTS acoustic model based on the predicted acoustic features that are aligned with the expected acoustic features, so that the improved TTS acoustic model can achieve better speech synthesis quality in fewer training iterations, thereby reducing the training difficulty and training cost, and providing a speech synthesis model with faster synthesis speed and higher synthesis effect, so as to improve the quality of synthesized speech.

[0099] See also Figure 4 , Figure 4 It is a schematic block diagram of a training device for a speech synthesis model provided in an embodiment of the present application. The training device for a speech synthesis model can be configured in a computer device to execute the aforementioned training method for the speech synthesis model.

[0100] like Figure 4 As shown, the training device 400 of the speech synthesis model includes: a data acquisition module 401, a model training module 402, a feature alignment module 403 and a parameter updating module 404.

[0101] A data acquisition module 401 is used to acquire a phoneme sequence of a text sample and real acoustic features of the speech of the text sample;

[0102] A model training module 402, used for inputting the phoneme sequence and the real acoustic features into an improved TTS acoustic model for training;

[0103] A feature alignment module 403 is used to guide the ordinary differential equation to perform acoustic feature generation processing on the phoneme sequence through the OT-CFM mechanism to obtain a predicted acoustic feature aligned with the expected acoustic feature of the text sample;

[0104] The parameter updating module 404 is used to update the parameters of the improved TTS acoustic model according to the predicted acoustic features and the real acoustic features until the improved TTS acoustic model converges to obtain a trained speech synthesis model, and the trained speech synthesis model is used to synthesize the speech of the text.

[0105] In one embodiment, the feature alignment module 403 is further used to:

[0106] By using the decoder of the improved TTS acoustic model, according to the ordinary differential equation, performing acoustic feature generation processing on the phoneme sequence to obtain the predicted acoustic features of the text sample;

[0107] Through the OT-CFM mechanism, the acoustic features of the phoneme sequence are calculated to obtain the expected acoustic features of the text sample, and the predicted acoustic features are aligned with the expected acoustic features to obtain the aligned predicted acoustic features.

[0108] In one embodiment, the feature alignment module 403 is further used to:

[0109] Calculating an OT-CFM loss function according to the predicted acoustic features and the target acoustic features;

[0110] According to the OT-CFM loss function, the predicted acoustic feature is aligned with the expected acoustic feature to obtain the aligned predicted acoustic feature.

[0111] In one embodiment, the feature alignment module 403 is further used to:

[0112] The OT-CFM loss function is minimized to obtain the aligned predicted acoustic features.

[0113] In one embodiment, the speech synthesis model training device 400 further includes a decoder conditioning module, wherein the decoder conditioning module is used to:

[0114] Obtaining a hidden state vector of the phoneme sequence through the encoder of the improved TTS acoustic model;

[0115] Predicting the duration of the phoneme sequence according to the hidden state vector by using the duration predictor of the improved TTS acoustic model;

[0116] The hidden state vector is sampled according to the duration to obtain a predicted average acoustic feature of the text sample, wherein the predicted average acoustic feature is used to condition the decoder.

[0117] In one embodiment, the decoder includes a convolutional neural network and a Transformer.

[0118] In one embodiment, the parameter updating module 404 is further used to:

[0119] Calculating a loss function between the aligned predicted acoustic features and the true acoustic features;

[0120] According to the loss function, the parameters of the improved TTS acoustic model are iteratively updated until the number of iterations is greater than a preset threshold, and it is determined that the improved TTS acoustic model converges to obtain the trained speech synthesis model.

[0121] Among them, each module in the above-mentioned speech synthesis model training device 400 corresponds to each step in the above-mentioned speech synthesis model training method embodiment, and its functions and implementation processes are no longer repeated here one by one.

[0122] The speech synthesis model training device 400 can execute the speech synthesis model training method provided in the embodiment of the present application. Therefore, it can achieve the beneficial effects that can be achieved by the speech synthesis model training method provided in the embodiment of the present application. Please refer to the previous embodiment for details and will not be repeated here.

[0123] The method and apparatus of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0124] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in Figure 5 Runs on the computer device shown.

[0125] See also Figure 5 , Figure 5 It is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application.

[0126] See also Figure 5 The computer device includes a processor and a memory connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0127] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0128] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any training method for the speech synthesis model.

[0129] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0130] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:

[0131] Acquire a phoneme sequence of a text sample and a real acoustic feature of the speech of the text sample;

[0132] Inputting the phoneme sequence and the real acoustic features into an improved text-to-speech (TTS) acoustic model for training;

[0133] By using the optimal transport condition flow matching (OT-CFM) mechanism, an ordinary differential equation is guided to perform acoustic feature generation processing on the phoneme sequence to obtain a predicted acoustic feature aligned with the expected acoustic feature of the text sample;

[0134] According to the aligned predicted acoustic features and the real acoustic features, the parameters of the improved TTS acoustic model are updated until the improved TTS acoustic model converges to obtain a trained speech synthesis model, and the trained speech synthesis model is used to synthesize the speech of the text.

[0135] In one embodiment, when the processor implements the process of guiding the ordinary differential equation to perform acoustic feature generation processing on the phoneme sequence through the OT-CFM mechanism to obtain the predicted acoustic feature aligned with the expected acoustic feature of the text sample, it is used to implement:

[0136] By using the decoder of the improved TTS acoustic model, according to the ordinary differential equation, performing acoustic feature generation processing on the phoneme sequence to obtain the predicted acoustic features of the text sample;

[0137] Through the OT-CFM mechanism, the acoustic features of the phoneme sequence are calculated to obtain the expected acoustic features of the text sample, and the predicted acoustic features are aligned with the expected acoustic features to obtain the aligned predicted acoustic features.

[0138] In one embodiment, when the processor implements the alignment processing of the predicted acoustic feature with the expected acoustic feature to obtain the aligned predicted acoustic feature, it is used to implement:

[0139] Calculating an OT-CFM loss function according to the predicted acoustic features and the target acoustic features;

[0140] According to the OT-CFM loss function, the predicted acoustic feature is aligned with the expected acoustic feature to obtain the aligned predicted acoustic feature.

[0141] In one embodiment, when the processor implements the alignment processing of the predicted acoustic feature with the expected acoustic feature according to the OT-CFM loss function to obtain the aligned predicted acoustic feature, it is used to implement:

[0142] The OT-CFM loss function is minimized to obtain the aligned predicted acoustic features.

[0143] In one embodiment, before implementing the decoder of the improved TTS acoustic model and performing acoustic feature generation processing on the phoneme sequence according to the ordinary differential equation to obtain the predicted acoustic features of the text sample, the processor is further used to implement:

[0144] Obtaining a hidden state vector of the phoneme sequence through the encoder of the improved TTS acoustic model;

[0145] Predicting the duration of the phoneme sequence according to the hidden state vector by using the duration predictor of the improved TTS acoustic model;

[0146] The hidden state vector is sampled according to the duration to obtain a predicted average acoustic feature of the text sample, wherein the predicted average acoustic feature is used to condition the decoder.

[0147] In one embodiment, the decoder includes a convolutional neural network and a Transformer.

[0148] In one embodiment, when the processor implements updating the parameters of the improved TTS acoustic model according to the aligned predicted acoustic features and the real acoustic features until the improved TTS acoustic model converges to obtain a trained speech synthesis model, it is used to implement:

[0149] Calculating a loss function between the aligned predicted acoustic features and the true acoustic features;

[0150] According to the loss function, the parameters of the improved TTS acoustic model are iteratively updated until the number of iterations is greater than a preset threshold, and it is determined that the improved TTS acoustic model converges to obtain the trained speech synthesis model.

[0151] The computer device can execute the training method of the speech synthesis model provided in the embodiment of the present application. Therefore, it can achieve the beneficial effects that can be achieved by the training method of the speech synthesis model provided in the embodiment of the present application. Please refer to the previous embodiment for details and will not be repeated here.

[0152] An embodiment of the present application also provides a computer-readable storage medium.

[0153] The computer-readable storage medium of the present application stores a computer program, and when the computer program is executed by a processor, the steps of the training method of the speech synthesis model as described above are implemented.

[0154] The computer-readable storage medium may be an internal storage unit of the training device of the speech synthesis model or the computer device described in the aforementioned embodiment, such as a hard disk or memory of the training device of the speech synthesis model or the computer device. The computer-readable storage medium may also be an external storage device of the training device of the speech synthesis model or the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital card (Secure Digital Card, SD Card), a flash card (Flash Card), etc., equipped on the training device of the speech synthesis model or the computer device.

[0155] Since the computer program stored in the computer-readable storage medium can execute any one of the speech synthesis model training methods provided in the embodiments of the present application, the beneficial effects that can be achieved by any one of the speech synthesis model training methods provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0156] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0157] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0158] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0159] The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and these modifications or substitutions should be included in the protection scope of the present application.

Claims

1. A method for training a speech synthesis model, characterized in that: The method comprises: Acquire a phoneme sequence of a text sample and a real acoustic feature of the speech of the text sample; Inputting the phoneme sequence and the real acoustic features into an improved text-to-speech (TTS) acoustic model for training; By using the optimal transport condition flow matching (OT-CFM) mechanism, an ordinary differential equation is guided to perform acoustic feature generation processing on the phoneme sequence to obtain a predicted acoustic feature aligned with the expected acoustic feature of the text sample; According to the aligned predicted acoustic features and the real acoustic features, the parameters of the improved TTS acoustic model are updated until the improved TTS acoustic model converges to obtain a trained speech synthesis model, and the trained speech synthesis model is used to synthesize the speech of the text.

2. The method according to claim 1, characterized in that The OT-CFM mechanism is used to guide the ordinary differential equation to perform acoustic feature generation processing on the phoneme sequence to obtain predicted acoustic features aligned with expected acoustic features of the text sample, including: By using the decoder of the improved TTS acoustic model, according to the ordinary differential equation, performing acoustic feature generation processing on the phoneme sequence to obtain the predicted acoustic features of the text sample; Through the OT-CFM mechanism, the acoustic features of the phoneme sequence are calculated to obtain the expected acoustic features of the text sample, and the predicted acoustic features are aligned with the expected acoustic features to obtain the aligned predicted acoustic features.

3. The method according to claim 2, characterized in that The aligning the predicted acoustic feature with the expected acoustic feature to obtain the aligned predicted acoustic feature includes: Calculating an OT-CFM loss function according to the predicted acoustic features and the target acoustic features; According to the OT-CFM loss function, the predicted acoustic feature is aligned with the expected acoustic feature to obtain the aligned predicted acoustic feature.

4. The method according to claim 3, characterized in that The step of aligning the predicted acoustic feature with the expected acoustic feature according to the OT-CFM loss function to obtain the aligned predicted acoustic feature comprises: The OT-CFM loss function is minimized to obtain the aligned predicted acoustic features.

5. The method according to claim 2, characterized in that: Before the decoder of the improved TTS acoustic model performs acoustic feature generation processing on the phoneme sequence according to the ordinary differential equation to obtain the predicted acoustic features of the text sample, the method further includes: Obtaining a hidden state vector of the phoneme sequence through the encoder of the improved TTS acoustic model; Predicting the duration of the phoneme sequence according to the hidden state vector by using the duration predictor of the improved TTS acoustic model; The hidden state vector is sampled according to the duration to obtain a predicted average acoustic feature of the text sample, wherein the predicted average acoustic feature is used to condition the decoder.

6. The method according to claim 2, characterized in that The decoder includes a convolutional neural network and a Transformer.

7. The method according to any one of claims 1 to 5, characterized in that: The step of updating parameters of the improved TTS acoustic model according to the aligned predicted acoustic features and the real acoustic features until the improved TTS acoustic model converges to obtain a trained speech synthesis model comprises: Calculating a loss function between the aligned predicted acoustic features and the true acoustic features; According to the loss function, the parameters of the improved TTS acoustic model are iteratively updated until the number of iterations is greater than a preset threshold, and it is determined that the improved TTS acoustic model converges to obtain the trained speech synthesis model.

8. A training device for a speech synthesis model, characterized in that: The device comprises: A data acquisition module, used to acquire a phoneme sequence of a text sample and real acoustic features of the speech of the text sample; A model training module, used for inputting the phoneme sequence and the real acoustic features into the TTS acoustic model for training; A feature alignment module, used to guide the ordinary differential equation to perform acoustic feature generation processing on the phoneme sequence through the OT-CFM mechanism, so as to obtain a predicted acoustic feature aligned with the expected acoustic feature of the text sample; A parameter updating module is used to update the parameters of the improved TTS acoustic model according to the predicted acoustic features and the real acoustic features until the improved TTS acoustic model converges to obtain a trained speech synthesis model, and the trained speech synthesis model is used to synthesize the speech of the text.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the training method of the speech synthesis model described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the training method of the speech synthesis model described in any one of claims 1 to 7 is implemented.

Images