Speech synthesis model training method, speech data acquisition method, device and apparatus

Abstract

The embodiment of the application discloses a speech synthesis model training method, a speech data acquisition method, an apparatus and equipment, and belongs to the technical field of computers. The method comprises the following steps: obtaining sample spectrum graphs and sample labels corresponding to sample speech data, obtaining first speech features corresponding to the sample spectrum graphs based on a feature extraction model; classifying the first speech features to obtain predicted labels, determining second gradient information opposite to first gradient information of a loss value of the present training with respect to the first speech features; generating predicted spectrum graphs based on a speech synthesis model and the first speech features; and training the speech synthesis model and the feature extraction model based on the sample spectrum graphs, the predicted spectrum graphs and the second gradient information. In the scheme provided by the embodiment of the application, the training method does not need to obtain high-quality and noise-free sample speech data, the requirement for the sample speech data is low, and the application range is wider.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a speech synthesis model training method, a speech data acquisition method, an apparatus, and a device. Background Technology

[0002] With the development of computer technology, speech synthesis technology is being applied more and more widely. For example, in intelligent customer service scenarios, voice broadcasting scenarios, or navigation scenarios, text information is often converted into a spectrogram based on a speech synthesis model, and then the spectrogram is converted into speech data for playback. Typically, the speech synthesis model needs to be trained before generating the spectrogram. Related technologies require high-quality, noise-free sample speech data to train the speech synthesis model; however, this method of training the speech synthesis model has high requirements for the sample speech data and a narrow range of applicability. Summary of the Invention

[0003] This application provides a speech synthesis model training method, speech data acquisition method, apparatus, and device, which can improve and reduce the requirements for sample speech data and expand the applicability of the training method.

[0004] The technical solution is as follows:

[0005] On the one hand, a method for training a speech synthesis model is provided, the method comprising:

[0006] Obtain the sample spectrogram and sample label corresponding to the sample speech data, wherein the sample label indicates whether the sample speech data contains noise;

[0007] Based on the feature extraction model, the first speech feature corresponding to the sample spectrogram is obtained; the first speech feature is classified to obtain a prediction label, and the prediction label indicates the possibility that the sample speech data contains noise.

[0008] Based on the first gradient information of the loss value of this training with respect to the first speech feature, a second gradient information that is inversely related to the first gradient information is determined, wherein the loss value is determined based on the sample label and the predicted label;

[0009] Based on the speech synthesis model and the first speech feature, a predicted spectrogram is generated;

[0010] The speech synthesis model and the feature extraction model are trained based on the sample spectrogram, the predicted spectrogram, and the second gradient information.

[0011] In one possible implementation, the method further includes:

[0012] Based on the speech synthesis model, the duration of the second phoneme feature information is predicted to obtain the predicted duration of each phoneme.

[0013] The step of training the speech synthesis model and the feature extraction model based on the sample spectrogram, the predicted spectrogram, and the second gradient information includes:

[0014] The speech synthesis model and the feature extraction model are trained based on the sample spectrogram, the predicted spectrogram, the second gradient information, the sample duration and the predicted duration of each phoneme.

[0015] In another possible implementation, classifying the first speech features to obtain predicted labels includes:

[0016] Based on the classification model, the first speech features are classified to obtain the predicted label;

[0017] The step of training the speech synthesis model and the feature extraction model based on the sample spectrogram, the predicted spectrogram, and the second gradient information includes:

[0018] The speech synthesis model, the feature extraction model, and the classification model are trained based on the sample spectrogram, the predicted spectrogram, and the second gradient information.

[0019] In another possible implementation, before training the speech synthesis model and the feature extraction model based on the sample spectrogram, the predicted spectrogram, and the second gradient information, the method further includes:

[0020] Obtain the sample spectrograms corresponding to sample speech data input from multiple accounts;

[0021] The speech synthesis model is trained based on the sample spectrograms corresponding to the multiple accounts.

[0022] On the other hand, a method for acquiring voice data is provided, the method comprising:

[0023] The model training interface is displayed, and the model training interface displays a first prompt message. The first prompt message is used to prompt authorization to collect the voice data input by the target account in the current application, and the collected voice data is used to train the target speech synthesis model corresponding to the target account.

[0024] In response to the confirmation operation of the first prompt information, the voice data collection permission of the target account is obtained, and the voice data collection permission indicates that the voice data input based on the target account in the current application is allowed to be used to train the target speech synthesis model;

[0025] If the voice data collection permission of the target account is obtained, the voice data input by the target account in the current application is obtained.

[0026] In one possible implementation, obtaining voice data collection permission for the target account in response to a confirmation operation of the first prompt information includes:

[0027] In response to the confirmation operation of the first prompt information, a second prompt information is displayed. The second prompt information includes preset text and is used to prompt the user to input voice data according to the preset text.

[0028] Acquire the input voice data;

[0029] If the acquired voice data matches the preset text, the voice data collection permission for the target account is obtained.

[0030] On the other hand, a speech synthesis model training device is provided, the device comprising:

[0031] The acquisition module is used to acquire the sample spectrogram and sample label corresponding to the sample speech data, wherein the sample label indicates whether the sample speech data contains noise;

[0032] The classification module is used to obtain the first speech feature corresponding to the sample spectrogram based on the feature extraction model; classify the first speech feature to obtain a predicted label, the predicted label indicating the possibility that the sample speech data contains noise;

[0033] The determination module is used to determine a second gradient information that is inversely related to the first gradient information based on the first gradient information of the loss value of this training with respect to the first speech feature, wherein the loss value is determined based on the sample label and the predicted label;

[0034] The generation module is used to generate a predicted spectrogram based on the speech synthesis model and the first speech features;

[0035] The training module is used to train the speech synthesis model and the feature extraction model based on the sample spectrogram, the predicted spectrogram, and the second gradient information.

[0036] In one possible implementation, the acquisition module is further configured to acquire the gradient function of the target loss function with respect to the first parameter, the target loss function including a second parameter representing the sample label and a third parameter representing the predicted label, the third parameter including the first parameter, the first parameter representing the first speech feature, and the gradient function representing the relationship between the first gradient information and the sample label, the predicted label and the first speech feature;

[0037] The determining module is further configured to determine the first gradient information based on the gradient function, the sample label, the predicted label, and the first speech feature.

[0038] In another possible implementation, the first speech feature includes multiple sample phoneme features; the generation module is used to determine similar phoneme features for each sample phoneme feature from multiple standard phoneme features contained in the phoneme mapping table; to construct a second speech feature from the similar phoneme features of the multiple sample phoneme features; and to generate the predicted spectrogram based on the speech synthesis model, the first speech feature and the second speech feature.

[0039] In another possible implementation, the phoneme mapping table is included in the phoneme clustering model, and the step of obtaining the second speech feature based on the first speech feature is implemented based on the phoneme clustering model; the training module is used to train the speech synthesis model, the feature extraction model, the phoneme clustering model, and the phoneme mapping table based on the sample spectrogram, the predicted spectrogram, and the second gradient information.

[0040] In another possible implementation, the generation module is used to process the phoneme sequence corresponding to the sample speech data and the first speech feature based on the speech synthesis model to obtain the predicted spectrogram, wherein the phoneme sequence includes at least one phoneme, and the pronunciation of the at least one phoneme matches the sound in the sample speech data.

[0041] In another possible implementation, the generation module is used to obtain first phoneme feature information corresponding to the phoneme sequence; fuse the first phoneme feature information with the first speech feature to obtain a fused feature; and perform speech synthesis on the fused feature to obtain the predicted spectrogram.

[0042] In another possible implementation, the generation module is used to extract features from the phoneme sequence to obtain second phoneme feature information, the second phoneme feature information including phoneme features corresponding to each phoneme in the phoneme sequence; according to the sample duration of each phoneme, the phoneme features corresponding to each phoneme in the second phoneme feature information are expanded to obtain first phoneme feature information, so that the number of phoneme features corresponding to each phoneme in the first phoneme feature information matches the corresponding sample duration, the sample duration indicating the duration of the pronunciation of the phoneme in the sample speech data.

[0043] In another possible implementation, the device further includes:

[0044] The prediction module is used to predict the duration of the second phoneme feature information based on the speech synthesis model, so as to obtain the predicted duration of each phoneme.

[0045] The training module is used to train the speech synthesis model and the feature extraction model based on the sample spectrogram, the predicted spectrogram, the second gradient information, the sample duration and the predicted duration of each phoneme.

[0046] In another possible implementation, the device further includes:

[0047] The recognition module is used to perform emotion recognition on the sample speech data to obtain the emotion type to which the sample speech data belongs;

[0048] The acquisition module is also used to acquire the emotional features corresponding to the emotional type;

[0049] The generation module is used to process the phoneme sequence, the emotion feature, and the first speech feature based on the speech synthesis model to obtain the predicted spectrogram.

[0050] In another possible implementation, the device further includes:

[0051] The acquisition module is also used to acquire target text information;

[0052] The mapping module is used to perform phoneme mapping on the target text information to obtain the target phoneme sequence corresponding to the target text information;

[0053] The processing module is used to process the target phoneme sequence based on the target speech synthesis model to obtain the target spectrogram;

[0054] The conversion module is used to perform speech conversion on the target spectrogram to obtain target speech data corresponding to the target text information.

[0055] In another possible implementation, the classification module is used to classify the first speech features based on a classification model to obtain the predicted label;

[0056] The training module is used to train the speech synthesis model, the feature extraction model, and the classification model based on the sample spectrogram, the predicted spectrogram, and the second gradient information.

[0057] In another possible implementation, the acquisition module is configured to, when the voice data acquisition permission for the target account has been obtained, acquire voice data input by the target account in the logged-in application, wherein the voice data acquisition permission indicates that the voice data input by the target account in the application is permitted to be used to train the target speech synthesis model corresponding to the target account; determine the acquired voice data as the sample voice data; and perform spectral conversion on the sample voice data to obtain the sample spectrogram.

[0058] In another possible implementation, the acquisition module is further configured to acquire a sample spectrogram corresponding to sample voice data input from multiple accounts;

[0059] The training module is also used to train the speech synthesis model based on the sample spectrograms corresponding to the multiple accounts.

[0060] On the other hand, a voice data acquisition device is provided, the device comprising:

[0061] The display module is used to display the model training interface. The model training interface displays a first prompt message. The first prompt message is used to prompt authorization to collect the voice data input by the target account in the current application, and the collected voice data is used to train the target speech synthesis model corresponding to the target account.

[0062] The acquisition module is used to acquire the voice data collection permission of the target account in response to the confirmation operation of the first prompt information. The voice data collection permission indicates that the voice data input based on the target account in the current application is allowed to be used to train the target speech synthesis model.

[0063] The acquisition module is further configured to acquire the voice data input by the target account in the current application, provided that the target account has been granted voice data acquisition permission.

[0064] In one possible implementation, the acquisition module is configured to, in response to a confirmation operation of the first prompt information, display a second prompt information, the second prompt information including preset text, the second prompt information being used to prompt the input of voice data according to the preset text; acquire the input voice data; and, if the acquired voice data matches the preset text, acquire the voice data collection permission of the target account.

[0065] In another possible implementation, the display module is further configured to display the status information of the target speech synthesis model in the model display interface, the status information indicating the training progress of the target speech synthesis model on multiple emotion types.

[0066] In another possible implementation, the model display interface includes a text input area and an emotion selection area; the device also includes:

[0067] The acquisition module is also configured to, in response to a speech synthesis instruction in the model display interface, acquire the text information input in the text input area and acquire the emotion type selected in the emotion selection area;

[0068] The generation module is used to generate speech data that matches the text information and belongs to the emotion type, based on the target speech synthesis model, the emotion type and the text information;

[0069] The playback module is used to play the audio data.

[0070] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to perform the operations performed by the speech synthesis model training method or speech data acquisition method as described above.

[0071] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to perform the operations performed by the speech synthesis model training method or speech data acquisition method as described above.

[0072] In another aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, performs the operations performed by the speech synthesis model training method or the speech data acquisition method as described above.

[0073] In the scheme provided in this application embodiment, adversarial training is performed by combining a feature extraction model and a speech synthesis model. The speech features obtained by the feature extraction model are used in the speech synthesis model. This process involves training not only based on the sample spectrogram and predicted spectrogram corresponding to the sample speech data, but also considering that the loss value determined based on the sample labels and predicted labels can reflect the accuracy of whether the speech features extracted by the feature extraction model contain noise. If the feature extraction model is trained based on the first gradient information of the speech features extracted by the feature extraction model with respect to this loss value, the speech features obtained by the feature extraction model can include noise-related features, making it easier to distinguish whether noise is present. Therefore, a method of inverting the gradient information is adopted, and training is performed based on the second gradient information, which is the inverse of the first gradient information. This makes the speech features obtained by the feature extraction model tend to not contain noise-related features, that is, the speech features obtained by the feature extraction model are not affected by noise. The speech features obtained by the feature extraction model are then used to train the speech synthesis model, enabling the trained speech synthesis model to generate noise-free spectrograms. This training method does not require high-quality and noise-free sample speech data, has low requirements for sample speech data, and has a wider range of applications. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 This is a schematic diagram of the structure of an implementation environment provided in an embodiment of this application;

[0076] Figure 2 This is a flowchart of a speech synthesis model training method provided in an embodiment of this application;

[0077] Figure 3 This is a flowchart of another speech synthesis model training method provided in the embodiments of this application;

[0078] Figure 4 This is a flowchart of obtaining a phoneme sequence provided in an embodiment of this application;

[0079] Figure 5 This is a flowchart illustrating a training method for a speech synthesis model and a feature extraction model, as provided in an embodiment of this application.

[0080] Figure 6 This is a flowchart illustrating a training speech synthesis model provided in an embodiment of this application;

[0081] Figure 7 This is a schematic diagram of an emotion type provided in an embodiment of this application;

[0082] Figure 8 This is a flowchart of another speech synthesis model training method provided in the embodiments of this application;

[0083] Figure 9 This is a flowchart of another speech synthesis model training method provided in the embodiments of this application;

[0084] Figure 10 This is a flowchart illustrating the generation of a predicted spectrogram based on multiple models, as provided in an embodiment of this application.

[0085] Figure 11 This is a flowchart of a voice data acquisition method provided in an embodiment of this application;

[0086] Figure 12 This is a flowchart of an authorization confirmation process provided in an embodiment of this application;

[0087] Figure 13 This is another flowchart for confirming authorization provided in an embodiment of this application;

[0088] Figure 14 This is a flowchart illustrating the generation of speech data based on a speech synthesis model, as provided in an embodiment of this application.

[0089] Figure 15 This is a schematic diagram of the structure of a speech synthesis model training device provided in an embodiment of this application;

[0090] Figure 16 This is a schematic diagram of another speech synthesis model training device provided in an embodiment of this application;

[0091] Figure 17 This is a schematic diagram of the structure of a voice data acquisition device provided in an embodiment of this application;

[0092] Figure 18 This is a schematic diagram of another voice data acquisition device provided in an embodiment of this application;

[0093] Figure 19 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application;

[0094] Figure 20 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0095] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0096] The terms “first,” “second,” “third,” etc., used in this application may be used to describe various concepts herein, but unless otherwise specified, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, without departing from the scope of this application, a first speech feature may be referred to as a second speech feature, and similarly, a second speech feature may be referred to as a first speech feature.

[0097] As used in this application, the terms "at least one," "multiple," "each," and "any" have the following meanings: at least one includes one, two, or more; multiple includes two or more; each refers to each of the corresponding multiple; and any refers to any one of the multiple. For example, multiple phonemes include three phonemes, where each refers to each of the three phonemes, and any refers to any one of the three phonemes, which could be the first phoneme, the second phoneme, or the third phoneme.

[0098] In the specific embodiments of this application, the voice data and other related data involved require user permission or consent when the above embodiments of this application are applied to specific products or technologies, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0099] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0100] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0101] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learn-by-doing.

[0102] The solution provided in this application embodiment is based on artificial intelligence machine learning technology, which can train a feature extraction model and a speech synthesis model, and generate speech data based on the trained speech synthesis model.

[0103] The speech synthesis model training method provided in this application is executed by a computer device. Optionally, the computer device is a terminal or a server. Additionally, the speech data acquisition method provided in this application is executed by a terminal.

[0104] Optionally, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, or in-vehicle terminal, but is not limited to these.

[0105] In some embodiments, the computer device is provided as a server. Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application. See also... Figure 1 The implementation environment includes terminal 101 and server 102. Terminal 101 and server 102 are connected via a wireless or wired network.

[0106] Terminal 101 has at least one application installed, which has the function of processing voice data. For example, a user can input voice data through the application to chat or make voice calls with others; or, a user can search for items based on the input voice data through the application. Terminal 101 can log in to the application based on a target account, enable the voice data collection permission of the target account, and collect voice data input by the target account in the authorized application when the voice data collection permission of the target account has been obtained. Server 102 is used to obtain the voice data collected by terminal 101 when the voice data collection permission of the target account has been obtained, and use the collected voice data to train the voice synthesis model corresponding to the target account, so that the voice data generated based on the trained target voice synthesis model is noise-free and the timbre of the voice data is the same as the timbre of the user to whom the target account belongs.

[0107] Figure 2 This is a flowchart illustrating a speech synthesis model training method provided in an embodiment of this application. The method is executed by a computer device, which may be a terminal or a server. Figure 2 As shown, the method includes:

[0108] 201. The computer device acquires the sample spectrogram and sample label corresponding to the sample speech data. The sample label indicates whether the sample speech data contains noise.

[0109] The sample spectrogram is a frequency distribution map of the sound in the sample speech data, used to characterize the distribution of sound frequencies in the sample speech data. The sample spectrogram can be of any form, such as a MEL (Meller-Alternate) spectrogram or other forms. The sample label can be represented in any form, such as a feature vector, or a fixed numerical value, like 0 or 1. A sample label of 1 indicates that the sample speech data contains noise, and a sample label of 0 indicates that the sample speech data does not contain noise.

[0110] 202. The computer device obtains the first speech feature corresponding to the spectrogram of the sample based on the feature extraction model; classifies the first speech feature to obtain a predicted label, which indicates the possibility that the sample speech data contains noise.

[0111] The feature extraction model is used to extract speech features from the spectrogram. This feature extraction model can be any network model, such as a GRU (Gated Recurrent Unit) or other network models. The first speech feature is used to characterize the sample speech data. This first speech feature can be represented in any form, such as a feature vector or a feature matrix. The predicted label can be represented in any form, such as a feature vector or a probability. A higher probability indicates a greater likelihood that the sample speech data contains noise, while a lower probability indicates a lower likelihood.

[0112] 203. The computer device determines a second gradient information that is the opposite of the first gradient information based on the first gradient information of the loss value of the first speech feature during the training. The loss value is determined based on the sample label and the predicted label.

[0113] In this embodiment, the predicted label is obtained by classifying the first speech feature, meaning the predicted label is related to the first speech feature. Therefore, the loss value determined based on the sample label and the predicted label is related to the first speech feature. Furthermore, since the first speech feature is obtained based on a feature extraction model, the first gradient information of the loss value with respect to the first speech feature is related to the feature extraction model. The loss value determined based on the sample label and the predicted label represents the degree of difference between the sample label and the predicted label, reflecting the accuracy of the speech features extracted by the feature extraction model in terms of noise. If the feature extraction model is trained based on the first gradient information of the loss value with respect to the speech features extracted by the feature extraction model, the speech features obtained by the feature extraction model can include noise-related features, making it easier to distinguish whether or not noise is present. Conversely, if the feature extraction model is trained based on the second gradient information, which is the opposite of the first gradient information, the speech features obtained by the feature extraction model can not include noise-related features, making it difficult to distinguish whether or not noise is present, even if the speech features obtained by the feature extraction model are unaffected by noise.

[0114] 204. The computer device generates a predicted spectrogram based on the speech synthesis model and the first speech feature.

[0115] The predicted spectrogram is the frequency distribution map of the sound in the predicted speech data of the sample. The speech synthesis model is used to generate the spectrogram. The speech synthesis model can be any network model, such as FastSpeech (a speech synthesis model) or Tacotron (a deep neural network model).

[0116] 205. The computer equipment trains the speech synthesis model and the feature extraction model based on the sample spectrogram, the predicted spectrogram, and the second gradient information.

[0117] In the scheme provided in this application embodiment, adversarial training is performed by combining a feature extraction model and a speech synthesis model. The speech features obtained by the feature extraction model are used in the speech synthesis model. This process involves training not only based on the sample spectrogram and predicted spectrogram corresponding to the sample speech data, but also considering that the loss value determined based on the sample labels and predicted labels can reflect the accuracy of whether the speech features extracted by the feature extraction model contain noise. If the feature extraction model is trained based on the first gradient information of the speech features extracted by the feature extraction model with respect to this loss value, the speech features obtained by the feature extraction model can include noise-related features, making it easier to distinguish whether noise is present. Therefore, a method of inverting the gradient information is adopted, and training is performed based on the second gradient information, which is the inverse of the first gradient information. This makes the speech features obtained by the feature extraction model tend to not contain noise-related features, that is, the speech features obtained by the feature extraction model are not affected by noise. The speech features obtained by the feature extraction model are then used to train the speech synthesis model, enabling the trained speech synthesis model to generate noise-free spectrograms. This training method does not require high-quality and noise-free sample speech data, has low requirements for sample speech data, and has a wider range of applications.

[0118] exist Figure 2 Based on the embodiment shown, when determining the first gradient information, the first parameter of the target loss function with respect to the first speech feature can be obtained. The specific process is detailed in the following embodiment.

[0119] Figure 3 This is a flowchart of a speech synthesis model training method provided in an embodiment of this application. The method is executed by a computer device, such as... Figure 3 As shown, the method includes:

[0120] 301. When a computer device has obtained the voice data collection permission for a target account, it acquires the voice data entered by the target account in the logged-in application and identifies the acquired voice data as the sample voice data corresponding to the target account.

[0121] The voice data collection permission indicator allows the use of voice data input from the target account within the application to train the target speech synthesis model corresponding to that target account. In other words, the application is authorized to collect voice data from the target account. This target speech synthesis model is a dedicated speech synthesis model to be trained for the target account. The timbre of the voice data obtained by converting the spectrogram generated by this target speech synthesis model is identical to the timbre of the user to whom the target account belongs.

[0122] In this embodiment, when voice collection permissions for the target account have been obtained, voice data input by the target account in authorized applications can be collected as sample voice data. This sample voice data is then used to train a voice synthesis model for the target account. For example, when a user logs into an authorized application using their account on a terminal and engages in voice calls or chats with friends through that application, the computer device, having obtained voice data collection permissions for the target account, will collect the voice data generated by the user during these calls or chats, using this data as sample voice data for subsequent training.

[0123] 302. The computer equipment performs spectrum conversion on the sample speech data to obtain the sample spectrum diagram corresponding to the sample speech data.

[0124] In one possible implementation, the process of obtaining the sample spectrogram includes: performing a Fourier transform on the sample speech data to transform the sample speech information from the time domain to the frequency domain, thereby obtaining the sample spectrogram.

[0125] In one possible implementation, the sample spectrogram is a Mel spectrogram. The process of obtaining the sample spectrogram includes: performing a Fourier transform on the sample speech data to obtain a first spectrogram; and performing Mel calculations on the first spectrogram to obtain the sample spectrogram. The Mel calculation transforms the frequencies in the first spectrogram into frequencies on a Mel scale, thereby obtaining the Mel spectrogram.

[0126] 303. The computer device acquires the sample label corresponding to the sample speech data. The sample label indicates whether the sample speech data contains noise.

[0127] In one possible implementation, the sample label includes a first sample label or a second sample label, wherein the first sample label indicates that the sample speech data contains noise, and the second sample label indicates that the sample speech data does not contain noise.

[0128] In this embodiment of the application, considering that the environment in which a user inputs voice data in an application is complex and diverse, the voice data input in the application is usually noisy. If the voice data input by the target account in the logged-in application is obtained as sample voice data, it is considered that the sample voice data is also noisy. Therefore, the sample label corresponding to the sample voice data is determined as the first sample label.

[0129] 304. The computer device obtains the first speech feature corresponding to the spectrogram of the sample based on the feature extraction model; classifies the first speech feature to obtain the predicted label.

[0130] In order to train the speech synthesis model, this application also introduces a feature extraction model, thereby combining the two models for adversarial training. Specifically, for the sample speech data used in training the speech synthesis model, the feature extraction model can extract speech features from the sample spectrogram corresponding to the sample speech data. This feature extraction model is the model to be trained; for example, it is an initialized model, or a model that has acquired feature extraction capabilities through preliminary training.

[0131] The prediction label indicates the probability that the sample speech data contains noise; that is, the prediction label indicates the probability that the sample speech data contains noise.

[0132] In one possible implementation, the first speech feature and the sample spectrogram satisfy the following relationship:

[0133] Z = GRU(x)

[0134] Where Z represents the first speech feature, GRU(·) represents the feature extraction model, and x represents the sample feature map.

[0135] In one possible implementation, the process of classifying the first speech feature includes: classifying the first speech feature based on a classification model to obtain the predicted label.

[0136] The classification model is used to classify speech features. The classification model can be any network model, such as a classification network with a gradient reversal layer (GRL).

[0137] Optionally, if the classification model includes a transformation sub-model and a classification sub-model, then the process of obtaining the predicted label based on the classification model includes: performing feature transformation on the first speech feature based on the transformation sub-model in the classification model to obtain the third speech feature; and classifying the third speech feature based on the classification sub-model in the classification model to obtain the predicted label.

[0138] The transformation sub-model is used to perform feature transformation on speech features. This transformation sub-model can be any network model; for example, it may include a linear transformation layer to perform a linear transformation on the speech features. The classification sub-model can be any audio classifier; for example, it may be a Softmax (logistic regression) classifier.

[0139] Optionally, the process of obtaining the predicted label based on the transformation sub-model and the classification sub-model in the classification model satisfies the following relationship:

[0140] Z′=W1Z+b

[0141]

[0142] Where Z represents the first speech feature, W1 is the constant transformation matrix in the transformation sub-model, b is the adjustment parameter in the transformation sub-model, and b is a constant; Z′ represents the third speech feature; W2 is the constant transformation matrix in the classification sub-model; and Softmax(·) represents the logistic regression function. This indicates the predicted label.

[0143] 305. The computer device obtains the gradient function of the target loss function with respect to the first parameter, the target loss function including a second parameter referring to the sample label and a third parameter referring to the predicted label, the third parameter including the first parameter, the first parameter referring to the first speech feature.

[0144] The target loss function can be any loss function, such as the cross-entropy loss function or other loss functions. The gradient function represents the relationship between the first gradient information and the sample label, the predicted label, and the first speech feature; the gradient function includes a second parameter, a third parameter, and a first parameter.

[0145] In one possible implementation, the target loss function satisfies the following relationship:

[0146]

[0147]

[0148] a′=W1a+b

[0149] Where l represents the loss value and x represents the second parameter. W1 is a constant transformation matrix, b is an adjustment parameter, b is a constant, and a is used to represent the first parameter.

[0150] In one possible implementation, the partial derivative of the loss function with respect to the first parameter is used to determine the gradient function. The gradient function is then as follows:

[0151]

[0152] Where l represents the loss value and a represents the first parameter. a is used to represent the third parameter, and a′ is used to represent the fourth parameter referring to the third speech feature; Used to represent the partial derivative of the loss value l with respect to the first parameter a, and used to refer to the first gradient information. Used to represent the loss value l with respect to the third parameter The partial derivatives, Used to represent the third parameter The partial derivative with respect to the fourth parameter a′, Used to represent the partial derivative of the fourth parameter a′ with respect to the first parameter a.

[0153] 306. The computer device determines the first gradient information of the loss value of this training with respect to the first speech feature based on the gradient function, the sample label, the predicted label, and the first speech feature.

[0154] In this training exercise, the loss value is determined based on the sample label, the predicted label, and the target loss function. This loss value represents the degree of difference between the sample label and the predicted label. In this embodiment, the gradient function includes a representation of the relationship between the first gradient information and the sample label, the predicted label, and the first speech feature. The gradient function includes a second parameter referring to the sample label and a third parameter referring to the predicted label. The third parameter includes a first parameter referring to the first speech feature. Substituting the sample label, the predicted label, and the first speech feature into the gradient function yields the first gradient information.

[0155] It should be noted that the embodiments of this application are illustrated by taking the acquisition of the first gradient information based on the target loss function as an example. In another embodiment, steps 305-307 do not need to be executed. Instead, other methods are adopted to determine the first gradient information of the loss value of this training with respect to the first speech feature.

[0156] 307. Based on the first gradient information of the loss value of this training with respect to the first speech feature, the computer device determines the second gradient information that is inversely related to the first gradient information.

[0157] In one possible implementation, the second gradient information satisfies the following relationship:

[0158]

[0159] Where l represents the loss value, and Z represents the first speech feature. Z′ represents the predicted label and is used to represent the third speech feature. The partial derivative of the loss value l with respect to the first speech feature Z is used to represent the second gradient information. Used to represent the loss value l against the predicted label The partial derivatives, Used to represent prediction labels The partial derivative with respect to the third speech feature Z′, is used to represent the partial derivative of the third speech feature Z' with respect to the first speech feature Z.

[0160] In a possible implementation, the above steps 305-307 are executed based on a gradient backpropagation model to obtain the second gradient information. The gradient extraction model is configured to acquire gradient information using a target loss function.

[0161] 308. A computer device processes the phoneme sequence corresponding to the sample speech data and the first speech feature based on a speech synthesis model to obtain the predicted spectrogram.

[0162] Wherein, the phoneme sequence comprises at least one phoneme, and the pronunciation of the at least one phoneme matches the sound in the sample speech data. In the embodiments of the present application, the phoneme sequence matches the sound in the sample speech data, and the first speech feature is obtained by processing the sample spectrogram using the feature extraction model. Then, the phoneme sequence and the first speech feature are processed based on the speech synthesis model, so that the obtained predicted spectrogram matches the sample speech data as much as possible.

[0163] In a possible implementation, as shown in Figure 4 , the process of acquiring the phoneme sequence comprises:

[0164] 401. Perform speech recognition on the sample speech data to obtain text information matching the sample speech data.

[0165] 402. Perform phoneme mapping on the text information to obtain the phoneme sequence.

[0166] In the embodiments of the present application, the recognition manner from speech data to text information and then from text information to phonemes is adopted to acquire the phoneme sequence corresponding to the sample speech data, which can ensure the accuracy of the recognized phoneme sequence as much as possible.

[0167] Optionally, an ASR (Automatic Speech Recognition) method is adopted to perform speech recognition on the sample speech data to obtain the text information.

[0168] Optionally, the phoneme sequence is a Pinyin sequence of the characters included in the text information. For example, if the text information obtained by performing speech recognition on the sample speech data is "who am I", the phoneme sequence obtained by performing phoneme mapping on the text information is "woshishui".

[0169] Optionally, the text information is a letter sequence comprising at least one letter, and the letter sequence is taken as the phoneme sequence. If the text information comprises a plurality of English words, the phoneme sequence is a sequence composed of the letters contained in the plurality of English words.

[0170] It should be noted that the embodiments of this application are based on a speech synthesis model, which processes the phoneme sequence and the first speech feature to obtain the predicted spectrogram. In another embodiment, it is not necessary to perform the above step 308. Instead, other methods are used to generate the predicted spectrogram based on the speech synthesis model and the first speech feature.

[0171] 309. The computer device trains the speech synthesis model and the feature extraction model based on the sample spectrogram, the predicted spectrogram, and the second gradient information.

[0172] In one possible implementation, where the predicted label is obtained based on a classification model, step 309 includes: training the speech synthesis model, the feature extraction model, and the classification model based on the sample spectrogram, the predicted spectrogram, and the second gradient information.

[0173] By using the sample spectrogram, the predicted spectrogram, and the second gradient information, adversarial training is conducted on the speech synthesis model, the feature extraction model, and the classification model. First, the classification accuracy of the classification model for speech features can be improved. In order to make the speech features obtained by the feature extraction model as indistinguishable as to whether they contain shapes, the speech features obtained by the feature extraction model tend to not contain noise-related features. During the training process, the speech synthesis model uses the speech features obtained by the feature extraction model to generate the predicted spectrogram, thereby enabling the trained speech synthesis model to generate a noise-free spectrogram. This achieves adversarial training of the speech synthesis model, the feature extraction model, and the classification model.

[0174] In one possible implementation, the loss value determined based on the sample label and the predicted label is called the first loss value. Step 309 includes: determining the second loss value for this training based on the sample spectrogram and the predicted spectrogram; determining the third gradient information of the second loss value with respect to the parameters of the speech synthesis model; weighting and fusing the third gradient information with the second gradient information to obtain the fused gradient information; and training the speech synthesis model and the feature extraction model based on the fused gradient information.

[0175] The second loss value represents the degree of difference between the sample spectrogram and the predicted spectrogram, thus reflecting the accuracy of the spectrogram generated by the speech synthesis model. Any loss function can be used to determine the second loss value, such as the mean squared error loss function or other loss functions. Since the predicted spectrogram is obtained based on the speech synthesis model and is related to the parameters of the speech synthesis model, the second loss value determined based on the sample spectrogram and the predicted spectrogram is also related to the parameters of the speech synthesis model.

[0176] Optionally, when training the speech synthesis model and the feature extraction model based on the first fusion gradient information, the parameters of the speech synthesis model and the parameters of the feature extraction model are adjusted based on the first fusion gradient information of this training.

[0177] It should be noted that the embodiments of this application employ a weighted fusion method of gradient information, using the first fused gradient information to train the speech synthesis model and the feature extraction model. In another embodiment, the speech synthesis model and the feature extraction model can also be trained separately based on the third and second gradient information. For example, when training the speech synthesis model and the feature extraction model based on the third and second gradient information, the second gradient information is used to adjust both the speech synthesis model and the feature extraction model, and then the third gradient information is used to adjust both again; or, the third gradient information is used to adjust both the speech synthesis model and the feature extraction model, and then the second gradient information is used to adjust both again.

[0178] Optionally, taking the adjustment of the feature extraction model based on the second gradient information as an example, the parameters of the feature extraction model before and after adjustment satisfy the following relationship:

[0179]

[0180] Where, θ′ gru θ is used to represent the adjusted parameters of the feature extraction model. gru The parameters used to represent the feature extraction model before adjustment are λ, where λ is the parameter adjustment step size and λ is a constant. Used to represent the second gradient information.

[0181] It should be noted that the embodiments in this application are only illustrative examples of training the speech synthesis model and the feature extraction model once. In another embodiment, following steps 301-309 above, the speech synthesis model and the feature extraction model can be trained multiple times using multiple sample speech data corresponding to the target account, until the number of training iterations reaches a threshold, or until the second loss value between the current training sample spectrogram and the predicted spectrogram is less than a loss value threshold. The number of training iterations threshold can be any value, for example, 20 or 30. The loss value threshold can be any value, for example, 0.3 or 0.2, etc.

[0182] It should be noted that in this embodiment, the voice data of the target account in the logged-in application is used as sample voice data after obtaining the target account's voice collection permission. In another embodiment, step 301 is not required; instead, other methods are used to obtain the sample voice data corresponding to the target account. In one possible implementation, other methods are used to obtain voice data recorded based on the target account, and the obtained voice data is used as sample voice data.

[0183] It should be noted that, in this embodiment, after obtaining voice collection permissions for the target account, the voice data of the target account in the logged-in application is used as sample voice data, and the corresponding sample spectrogram is obtained. This spectrogram is then used to convert the generated speech data based on the spectrogram of the trained speech synthesis model. In another embodiment, steps 301-302 are not required; instead, other methods are used to obtain the sample spectrogram corresponding to the sample voice data. In one possible implementation, sample voice data corresponding to multiple accounts is obtained, and the corresponding sample spectrograms of the multiple sample voice data are obtained. These multiple sample spectrograms are then used to train the speech synthesis model to improve the ability of the speech synthesis model to generate noise-free spectrograms.

[0184] In the scheme provided in this application embodiment, adversarial training is performed by combining a feature extraction model and a speech synthesis model. The speech features obtained by the feature extraction model are used in the speech synthesis model. This process involves training not only based on the sample spectrogram and predicted spectrogram corresponding to the sample speech data, but also considering that the loss value determined based on the sample labels and predicted labels can reflect the accuracy of whether the speech features extracted by the feature extraction model contain noise. If the feature extraction model is trained based on the first gradient information of the speech features extracted by the feature extraction model with respect to this loss value, the speech features obtained by the feature extraction model can include noise-related features, making it easier to distinguish whether noise is present. Therefore, a method of inverting the gradient information is adopted, and training is performed based on the second gradient information, which is the inverse of the first gradient information. This makes the speech features obtained by the feature extraction model tend to not contain noise-related features, that is, the speech features obtained by the feature extraction model are not affected by noise. The speech features obtained by the feature extraction model are then used to train the speech synthesis model, enabling the trained speech synthesis model to generate noise-free spectrograms. This training method does not require high-quality and noise-free sample speech data, has low requirements for sample speech data, and has a wider range of applications.

[0185] Furthermore, voice data based on the input of the target account is collected from the target application authorized by the target account as sample voice data, and the voice synthesis model is trained according to the sample voice data to obtain the target voice synthesis model corresponding to the target account. The voice data obtained by converting the spectrogram generated by the target voice synthesis model is the same as the voice of the user to which the target account belongs, thus realizing the solution of generating a unique voice synthesis model for the target account.

[0186] Furthermore, by utilizing the target loss function, the gradient function of the target loss function with respect to the first parameter is first obtained, and then the second gradient information is obtained based on the gradient function to ensure the accuracy of the obtained second gradient information, so as to ensure the training effect of the speech synthesis model when training the model based on the second gradient information.

[0187] Furthermore, by using the sample spectrogram, the predicted spectrogram, and the second gradient information, adversarial training of the speech synthesis model, feature extraction model, and classification model can be achieved. First, the classification accuracy of the classification model for speech features can be improved. In order to make the speech features obtained based on the feature extraction model as indistinguishable as to whether they contain shapes, the speech features obtained based on the feature extraction model tend to not contain noise-related features. During the training process, the speech synthesis model uses the speech features obtained by the feature extraction model to generate the predicted spectrogram, thereby enabling the trained speech synthesis model to generate a noise-free spectrogram. This achieves adversarial training of the speech synthesis model, feature extraction model, and classification model.

[0188] In the above Figure 3 Based on the illustrated embodiment, and using a speech synthesis model, a predicted spectrogram can be generated by combining the sample duration of phonemes in the phoneme sequence, and the speech synthesis model and feature extraction model can be trained. For example... Figure 5 As shown, the process of generating a predicted spectrogram based on a speech synthesis model and training the speech synthesis model and feature extraction model includes:

[0189] 501. The computer device extracts features from the phoneme sequence based on the speech synthesis model to obtain the second phoneme feature information.

[0190] The second phoneme feature information is used to characterize the phoneme sequence, and includes the phoneme feature corresponding to each phoneme in the phoneme sequence. For example, if the phoneme sequence includes 10 phonemes, the second phoneme feature information includes the phoneme feature corresponding to each phoneme in the phoneme sequence, that is, the second phoneme feature information includes 10 phoneme features. The phoneme feature can be represented in any form; for example, the phoneme feature can be represented in the form of a feature vector.

[0191] In one possible implementation, step 501 includes: encoding each phoneme in the phoneme sequence based on a speech synthesis model to obtain phoneme features corresponding to each phoneme, and constructing the second phoneme feature information from the phoneme features corresponding to each phoneme.

[0192] 502. The computer device, based on a speech synthesis model, expands the phoneme features corresponding to each phoneme in the second phoneme feature information according to the sample duration of each phoneme in the phoneme sequence, so as to obtain the first phoneme feature information, so that the number of phoneme features corresponding to each phoneme in the first phoneme feature information matches the corresponding sample duration.

[0193] The sample duration of a phoneme indicates the duration of its pronunciation within the sample speech data. In this embodiment, the sum of the sample durations of multiple phonemes in the phoneme sequence represents the duration of the sound in the sample speech data. Each phoneme feature in the second phoneme feature information is expanded according to its sample duration, so that the number of phoneme features corresponding to each phoneme in the expanded first phoneme feature information matches the corresponding sample duration. This ensures that a predicted spectrogram matching the duration of the sound in the sample speech data can be generated subsequently based on the first phoneme feature information.

[0194] In one possible implementation, phoneme features correspond to unit phoneme duration. The process of obtaining the first phoneme feature information includes: for each phoneme in the second phoneme feature information, determining the ratio between the sample duration of the phoneme and the unit phoneme duration, expanding the phoneme features corresponding to the phoneme so that the number of expanded phoneme features is equal to the ratio, and constructing the first phoneme feature information from the expanded phoneme features corresponding to each phoneme.

[0195] For example, if the duration of a single phoneme corresponding to a phoneme feature is 0.1 seconds, and the phoneme sequence includes phoneme 1, phoneme 2, and phoneme 3, and the second phoneme feature information includes phoneme feature 1, phoneme feature 2, and phoneme feature 3, where phoneme 1 corresponds to phoneme feature 1, phoneme 2 corresponds to phoneme feature 2, and phoneme 3 corresponds to phoneme feature 3. If the sample duration of phoneme 1 is 0.5 seconds, the sample duration of phoneme 2 is 0.3 seconds, and the sample duration of phoneme 3 is 0.2 seconds, then after expanding the second phoneme feature information, the expanded first phoneme feature information includes 5 phoneme feature 1s, 3 phoneme feature 2s, and 2 phoneme feature 3s.

[0196] Optionally, the order of different phoneme features in the expanded first phoneme feature information is the same as the order of phoneme features in the second phoneme feature information. For example, the second phoneme feature information includes phoneme feature 1, phoneme feature 2, and phoneme feature 3, and the expanded first phoneme feature information includes phoneme feature 1, phoneme feature 2, phoneme feature 3, and phoneme feature 4. That is, the order of different phoneme features in the first phoneme feature information is phoneme feature 1, phoneme feature 2, and phoneme feature 3.

[0197] It should be noted that in this embodiment, the first phoneme feature information is obtained by directly expanding the second phoneme feature information after obtaining the second phoneme feature information. In another embodiment, after obtaining the second phoneme feature information, the phoneme features in the second phoneme feature information are updated based on the speech synthesis model using a self-attention mechanism. Then, the updated second phoneme feature information is expanded to obtain the first phoneme feature information.

[0198] In one possible implementation, the process of updating the second phoneme feature information includes: for the first phoneme feature in the second phoneme feature information, weighted fusion of the multiple phoneme features in the phoneme feature information and the first phoneme feature based on the similarity between the first phoneme feature and the multiple phoneme features in the phoneme feature information; fusion of the fused phoneme feature with the first phoneme feature to obtain the updated first phoneme feature; and using the updated phoneme feature of each phoneme feature in the second phoneme feature information to constitute the updated second phoneme feature information.

[0199] In this context, the first phoneme feature is any one of the multiple phoneme features included in the second phoneme feature information. Following the method described above for obtaining the updated first phoneme feature, the updated phoneme features for each phoneme feature in the second phoneme feature information can be obtained. By employing a self-attention mechanism to update the second phoneme feature information, each phoneme feature in the updated second phoneme feature information incorporates other phoneme features, enhancing the connections between the phoneme features in the updated second phoneme feature information and ensuring the accuracy of the updated second phoneme feature information.

[0200] It should be noted that in this embodiment, the method of first obtaining the second phoneme feature information and then expanding it is adopted to obtain the first phoneme feature information. In another embodiment, it is not necessary to perform the above steps 501-502, but other methods are adopted to obtain the first phoneme feature information corresponding to the phoneme sequence.

[0201] 503. The computer device, based on the speech synthesis model, fuses the first phoneme feature information with the first speech feature to obtain the fused feature.

[0202] For example, taking the first phoneme feature information and the first speech feature as both feature matrices, the first phoneme feature information is added to the first speech feature to obtain the fused feature, that is, the fused feature is also a feature matrix.

[0203] 504. The computer device uses a speech synthesis model to synthesize speech based on the fused features, and obtains the predicted spectrogram.

[0204] In this embodiment, the fused feature incorporates phoneme feature information and a first speech feature. After obtaining the fused feature, a predicted spectrogram is generated based on the speech synthesis model and the fused feature. The generated predicted spectrogram is influenced by the phoneme feature information and the first speech feature. When the first speech feature based on the feature extraction model tends not to contain noise-related features, and the phoneme feature information contains standard phoneme features, the predicted spectrogram tends to be unaffected by noise, and the predicted spectrogram matches the standard phoneme features, resulting in a more accurate predicted spectrogram.

[0205] In one possible implementation, the fused features are linearly transformed based on the speech synthesis model to obtain the predicted spectrogram. In this embodiment, linear transformation is used as a speech synthesis method to synthesize speech from the fused features.

[0206] It should be noted that in this embodiment, after obtaining the fusion features, speech synthesis is directly performed on the fusion features. In another embodiment, after obtaining the fusion features, a self-attention mechanism is first adopted to update the fusion features based on the speech synthesis model, and then speech synthesis is performed on the updated fusion features to obtain the predicted spectrogram.

[0207] In one possible implementation, the fusion feature includes multiple sub-features; then the process of updating the fusion feature includes: for the first sub-feature in the fusion feature, performing weighted fusion on the multiple sub-features based on the similarity between the first sub-feature and the multiple sub-features in the fusion feature, fusing the fused sub-features with the first sub-feature to obtain the updated first sub-feature, and using the updated sub-features of each sub-feature in the fusion feature to constitute the updated fusion feature.

[0208] The first sub-feature is any one of the multiple sub-features included in the fused feature. Following the method described above for obtaining the updated first sub-feature, the updated sub-features of each sub-feature in the fused feature can be obtained. By employing a self-attention mechanism to update the fused feature, each sub-feature in the updated fused feature incorporates other sub-features, strengthening the connections between the sub-features in the updated fused feature and ensuring the accuracy of the updated fused feature.

[0209] 505. Based on the speech synthesis model, the computer device performs duration prediction on the feature information of the second phoneme to obtain the predicted duration of each phoneme.

[0210] The predicted duration of a phoneme indicates the predicted duration of the pronunciation of that phoneme in the sample speech data.

[0211] In one possible implementation, based on the speech synthesis model, the duration of the second phoneme feature information is predicted to obtain predicted duration information. The predicted duration information includes at least one predicted duration, and each predicted duration is the predicted duration of a phoneme in the phoneme sequence. In the predicted duration information, any predicted duration corresponds to a phoneme with the same order in the phoneme sequence.

[0212] For example, the predicted duration information includes 10 predicted durations, the phoneme sequence includes 10 phonemes, the first predicted duration in the predicted duration information is the predicted duration of the phoneme sequence including the first phoneme, the second predicted duration in the predicted duration information is the predicted duration of the phoneme sequence including the second phoneme, and so on, so that the predicted duration of each phoneme in the phoneme sequence can be obtained.

[0213] It should be noted that the embodiments of this application are illustrated by first executing the above steps 502-504 and then executing the above steps 505. In another embodiment, the above steps 505 can be executed first and then the above steps 502-504 can be executed. This application does not limit this.

[0214] 506. The computer device trains the speech synthesis model and the feature extraction model based on the sample spectrogram, the predicted spectrogram, the second gradient information, the sample duration and the predicted duration of each phoneme.

[0215] The difference between the sample duration and the predicted duration of the same phoneme can reflect the accuracy of the duration obtained by the speech synthesis model.

[0216] In one possible implementation, the loss value determined based on the sample label and the predicted label is referred to as the first loss value. Step 309 includes: determining the second loss value based on the sample spectrogram and the predicted spectrogram; determining the third gradient information of the second loss value in this training with respect to the parameters of the speech synthesis model; determining the third loss value based on the sample duration and the predicted duration of each phoneme; determining the fourth gradient information of the third loss value in this training with respect to the parameters of the speech synthesis model; weightedly fusing the third gradient information, the fourth gradient information and the second gradient information to obtain the second fused gradient information; and training the speech synthesis model and the feature extraction model based on the second fused gradient information.

[0217] In the solution provided in this application, a speech synthesis model and a feature extraction model are trained based on the sample spectrogram, the predicted spectrogram, the second gradient information, the sample duration and predicted duration of each phoneme. Since the speech features obtained by the feature extraction model tend to be unaffected by noise, when the speech synthesis model generates the predicted spectrogram, it can learn speech features that tend to be unaffected by noise. This enables the trained speech synthesis model to generate spectrograms that tend to be noise-free. Furthermore, it improves the accuracy of the phoneme duration obtained by the speech synthesis model, ensuring the accuracy of the generated spectrogram when the speech synthesis model subsequently expands the phoneme feature information according to the predicted speech duration. It should be noted that this application embodiment uses a weighted fusion method of gradient information, training the speech synthesis model and the feature extraction model based on the fused second gradient information. In another embodiment, the speech synthesis model and the feature extraction model can be trained based on the third gradient information, the fourth gradient information, and the second gradient information, respectively.

[0218] It should be noted that the above Figure 5 The embodiment shown obtains the predicted spectrogram according to steps 501-504 above, and then trains the feature extraction model and speech synthesis model according to steps 505-506 above. In another embodiment, after performing steps 501-504 above, it is not necessary to perform steps 505-506 above. Instead, the feature extraction model and speech synthesis model are trained according to step 307 above.

[0219] It should be noted that the above Figure 3 and Figure 5 The embodiment shown directly uses the sample speech data corresponding to the target account to train the speech synthesis model. In another embodiment, the speech synthesis model needs to be pre-trained before using the sample speech data corresponding to the target account to train the speech synthesis model.

[0220] In one possible implementation, the process of pre-training the speech synthesis model includes: obtaining sample spectrograms corresponding to sample speech data input from multiple accounts; and training the speech synthesis model based on the sample spectrograms corresponding to the multiple accounts.

[0221] In this embodiment, considering the limited number of sample speech data corresponding to a single account, sample speech data from multiple accounts is first acquired. Based on the spectrograms of these sample speech data from multiple accounts, a speech synthesis model is trained to improve its ability to generate spectrograms. Then, the pre-trained speech synthesis model is trained again based on the sample speech data corresponding to the target account. This ensures that even with limited sample speech data for the target account, the speech data generated from the spectrogram of the trained speech synthesis model is sufficiently similar to the voice of the user belonging to the target account, guaranteeing the accuracy of the trained speech synthesis model. This eliminates the need to train the speech synthesis model with a large amount of sample speech data from the target account, thus expanding the applicability of this training method.

[0222] The process of training a speech synthesis model based on sample speech data corresponding to multiple accounts, as described above, is similar to the process described above. Figure 3 Similarly, the illustrated embodiment trains the speech synthesis model based on the sample spectrograms corresponding to the sample speech data of the same account each time. However, the difference lies in that during the training of the speech synthesis model based on sample speech data from multiple accounts, the account features are also incorporated when generating the predicted spectrogram based on the speech synthesis model. That is, the process of obtaining the predicted spectrogram based on the speech synthesis model includes: for the account corresponding to the sample speech data used in the current training, obtaining the account features corresponding to that account; and processing the phoneme sequence corresponding to the sample speech data, the account features corresponding to that account, and the first speech feature based on the speech synthesis model to obtain the predicted spectrogram. In one possible implementation, based on the speech synthesis model, following step 501 above, second phoneme feature information is obtained; this second phoneme feature information is fused with the account features to obtain updated second phoneme feature information; then, following steps 502-504 above, the predicted spectrogram is obtained.

[0223] like Figure 6 As shown, the process of training the speech synthesis model includes: 601. Pre-training the speech synthesis model based on speech data corresponding to multiple accounts; 602. Training the pre-trained speech synthesis model based on the speech data corresponding to the target account to obtain the target speech synthesis model corresponding to the target account.

[0224] In the solution provided in this application embodiment, when training the speech synthesis model based on the sample spectrograms corresponding to sample speech data of multiple accounts, the account features corresponding to the accounts are fused so that the speech synthesis model can distinguish sample speech data belonging to different accounts, thereby ensuring the accuracy of the speech synthesis model.

[0225] Furthermore, when training the speech synthesis model, the emotion type of the sample speech data can also be considered. In one possible implementation, emotion recognition is performed on the sample speech data to obtain the emotion type of the sample speech data, and the emotion features corresponding to the emotion type are obtained. Based on the speech synthesis model, after obtaining the second phoneme feature information according to step 501 above, the second phoneme feature information is fused with the emotion features to obtain updated second phoneme feature information. Then, based on the updated second phoneme features, the predicted spectrogram is obtained according to steps 502-504 above.

[0226] This emotion type includes multiple categories. Optionally, emotion types can be divided into multi-level emotion types according to different granularity levels. For example... Figure 7 As shown, emotion types include three levels. Level 1 emotion type 701, Level 2 emotion type 702, and Level 3 emotion type 703 each include multiple types. Level 3 emotion type 703 belongs to Level 2 emotion type 702, and Level 2 emotion type 702 belongs to Level 1 emotion type 701. Following the above method, sample speech data belonging to multiple emotion types are obtained. Based on the sample speech data belonging to multiple emotion types, according to the above... Figure 3 and Figure 5 The embodiment shown trains a speech synthesis model so that the trained speech synthesis model has the function of generating speech data of the user of the target account in each of a variety of emotions.

[0227] Optionally, the speech synthesis model includes multiple speech synthesis sub-models, each used to generate speech data belonging to a specific emotion type. Different speech synthesis sub-models correspond to different emotion types. In this embodiment, when training the speech synthesis model, sample speech data belonging to multiple emotion types are used to train the speech synthesis sub-models separately, thereby obtaining multiple speech synthesis sub-models.

[0228] It should be noted that the present application embodiment obtains the predicted spectrogram by fusing emotional features and second phoneme feature information. In another embodiment, the above process is not required. Instead, other methods are used to process the phoneme sequence, emotional features and first speech features based on the speech synthesis model to obtain the predicted spectrogram.

[0229] In the above Figure 2 Based on the illustrated embodiment, a phoneme mapping table is also introduced. This table determines similar and standard phoneme features for the sample phoneme features, and these standard phoneme features are also incorporated into the model training process. Figure 8 As shown, the speech synthesis model training method includes:

[0230] 801. The computer device acquires the sample spectrogram and sample label corresponding to the sample speech data. The sample label indicates whether the sample speech data contains noise.

[0231] 802. The computer device obtains the first speech feature corresponding to the spectrogram of the sample based on the feature extraction model; classifies the first speech feature to obtain a predicted label, which indicates the possibility that the sample speech data contains noise.

[0232] 803. Based on the first gradient information of the loss value of this training with respect to the first speech feature, the computer device determines the second gradient information that is inversely related to the first gradient information.

[0233] Steps 801-803 are the same as steps 201-203 above, and will not be repeated here.

[0234] 804. The computer device, based on a phoneme clustering model, determines the similar phoneme features of each sample phoneme feature in the first speech feature from multiple standard phoneme features contained in the phoneme mapping table.

[0235] The phoneme clustering model can be any network model. In this embodiment, the phoneme mapping table is included in the phoneme clustering model, and the first speech feature includes multiple sample phoneme features. The phoneme mapping table includes multiple standard phoneme features, each of which represents a phoneme. These multiple standard phoneme features are discretely distributed in the feature space, meaning they differ from each other and can distinguish multiple different phonemes. For any sample phoneme feature in the first speech feature, the similar phoneme features are the standard phoneme features included in the phoneme mapping table.

[0236] In one possible implementation, step 804 includes: for each sample phoneme feature in the first speech feature, determining the similarity between the sample phoneme feature and each standard phoneme feature contained in the phoneme mapping table, and determining the standard phoneme feature with the highest similarity among the multiple standard phoneme features contained in the phoneme mapping table as the similar phoneme feature of the sample phoneme feature.

[0237] For multiple sample phoneme features in the first speech feature, the similar phoneme features of each sample phoneme feature can be obtained by following the above steps. The similarity between the sample phoneme features and the standard phoneme features can be determined using cosine similarity or other similarity methods.

[0238] In the solution provided in this application embodiment, a similarity method is adopted to determine the similar phoneme features of each sample phoneme feature in the first speech feature from multiple standard phoneme features contained in the phoneme mapping table, so as to ensure the accuracy of the determined similar phoneme features.

[0239] 805. The computer device uses a phoneme clustering model to construct a second speech feature from the similar phoneme features of multiple sample phoneme features in the first speech feature.

[0240] After determining the similar phoneme features of each sample phoneme feature in the first speech feature, the determined similar phoneme features are used to construct the second speech feature. That is, the second speech feature contains the same number of phoneme features as the first speech feature, and all phoneme features in the second speech feature belong to the phoneme mapping table. Since the first speech feature is obtained by extracting features from sample speech data based on a feature extraction model, the differences between the sample phoneme features used to represent different phonemes in the first speech feature may be small, and the sample phoneme features contained in the first speech feature may be inaccurate. Therefore, the first speech feature is updated through a phoneme clustering model and the phoneme mapping table contained in the phoneme clustering model, so that the phoneme features contained in the obtained second speech feature are accurate, thereby ensuring the accuracy of the second speech feature.

[0241] In one possible implementation, the order of the multiple similar phoneme features included in the second speech feature is the same as the order of the sample phoneme features corresponding to the multiple similar phoneme features in the first speech feature.

[0242] In one possible implementation, the phoneme clustering model includes an encoder and a phoneme mapping table. Then, step 804 includes: encoding the first speech feature based on the encoder in the phoneme clustering model to obtain the fourth speech feature; determining the similar phoneme features of each sample phoneme feature in the fourth speech feature from multiple standard phoneme features contained in the phoneme mapping table based on the phoneme clustering model; and constructing the second speech feature from the similar phoneme features of multiple sample phoneme features in the fourth speech feature.

[0243] The encoder can encode the first speech feature by performing a linear transformation, that is, by performing a linear transformation on each sample phoneme feature in the first speech feature to obtain the fourth speech feature. The encoder can be of any form, for example, it can be a VQ-VAE (Vector Quatization-Variational Auto Encoder).

[0244] Optionally, the process of obtaining the fourth speech feature based on the encoder satisfies the following relationship:

[0245] h e (Z)=W emb Z+d

[0246] Among them, h e (Z) is used to represent the fourth speech feature; Wemb Z is the constant transformation matrix in the encoder, where Z represents the first speech feature and d represents the adjustment parameter in the encoder, where d is a constant.

[0247] It should be noted that the embodiment of this application takes the phoneme mapping table contained in the phoneme clustering model as an example, and obtains the second speech feature based on the phoneme clustering model. In another embodiment, it is not necessary to perform the above steps 804-805. Instead, other methods are adopted to determine the similar phoneme features of each sample phoneme feature from the multiple standard phoneme features contained in the phoneme mapping table; and the similar phoneme features of multiple sample phoneme features in the first speech feature constitute the second speech feature.

[0248] 806. The computer device generates the predicted spectrogram based on the speech synthesis model, the first speech feature, and the second speech feature.

[0249] Because the speech synthesis model incorporates a first speech feature and a second speech feature when generating the predicted spectrogram, the first speech feature is used to characterize the sample speech data, and the second speech feature is a feature obtained by standardizing and updating the first speech feature, the speech synthesis model can generate an accurate predicted spectrogram, thus ensuring the accuracy of the predicted spectrogram.

[0250] In one possible implementation, step 806 includes: processing the phoneme sequence, the first speech feature, and the second speech feature corresponding to the sample speech data based on the speech synthesis model to obtain the predicted spectrogram.

[0251] Optionally, based on the speech synthesis model, features are extracted from the phoneme sequence to obtain second phoneme feature information; based on the speech synthesis model, the phoneme features corresponding to each phoneme in the second phoneme feature information are expanded according to the sample duration of each phoneme in the phoneme sequence to obtain the first phoneme feature information, so that the number of phoneme features corresponding to each phoneme in the first phoneme feature information matches the corresponding sample duration; based on the speech synthesis model, the first phoneme feature information is fused with the first speech feature to obtain a first fused feature; the first fused feature is fused with the second speech feature to obtain a second fused feature; based on the speech synthesis model, speech is synthesized from the second fused feature to obtain the predicted spectrogram.

[0252] In this embodiment, the fusion feature obtained in step 503 is referred to as the first fusion feature. The process of obtaining the first fusion feature based on the speech synthesis model is the same as steps 501-503 above, and will not be repeated here. The process of performing speech synthesis on the second fusion feature based on the speech synthesis model is the same as step 504 above, and will not be repeated here.

[0253] For the method of fusing the first fusion feature and the second speech feature, taking the first fusion feature and the second speech feature as both being feature matrices as an example, the first fusion feature and the second speech feature are fused and added together to obtain the second fusion feature, that is, the second fusion feature is also a feature matrix.

[0254] Optionally, before fusing the first fusion feature with the second speech feature, a self-attention mechanism needs to be applied to update the first fusion feature, and then the updated first fusion feature is fused with the second speech feature. The process of applying the self-attention mechanism to update the first fusion feature is the same as the process of applying the self-attention mechanism to update the fusion feature in step 504 above, and will not be described again here.

[0255] It should be noted that the embodiment of this application takes updating the first speech feature using a phoneme mapping table as an example, and generates a predicted spectrogram based on the speech synthesis model, the first speech feature, and the updated second speech feature. In another embodiment, steps 804-806 are not required, but other methods are adopted to generate a predicted spectrogram based on the speech synthesis model and the first speech feature.

[0256] 807. The computer device trains the speech synthesis model, the feature extraction model, the phoneme clustering model, and the phoneme mapping table based on the sample spectrogram, the predicted spectrogram, and the second gradient information.

[0257] In this embodiment, the speech synthesis model, the feature extraction model, the phoneme clustering model, and the phoneme mapping table are trained based on the sample spectrogram, the predicted spectrogram, and the second gradient information. This ensures that the feature extraction model and the speech synthesis model are less affected by noise, while also improving the accuracy of the phoneme clustering model and the phoneme mapping table. Consequently, the second feature extraction model, updated based on the phoneme clustering model and the phoneme mapping table, obtains a more accurate spectrogram from the speech synthesis model, further enhancing the accuracy of the speech synthesis model.

[0258] In one possible implementation, the predicted spectrogram generated based on the speech synthesis model is referred to as the first spectrogram. The method further includes: processing the second speech feature based on the phoneme clustering model to obtain the second predicted spectrogram. Then, step 807 includes: training the speech synthesis model, the feature extraction model, the phoneme clustering model, and the phoneme mapping table based on the difference between the sample spectrogram and the first predicted spectrogram, the difference between the sample spectrogram and the second predicted spectrogram, and the second gradient information.

[0259] The difference between the sample spectrogram and the second predicted spectrogram reflects the accuracy of the phoneme clustering model and the phoneme mapping table. Based on the differences between the sample spectrogram and the first predicted spectrogram, the differences between the sample spectrogram and the second predicted spectrogram, and the second gradient information, the speech synthesis model, the feature extraction model, the phoneme clustering model, and the phoneme mapping table are trained. This ensures that the feature extraction model and the speech synthesis model are less affected by noise, while also improving the accuracy of the phoneme clustering model and the phoneme mapping table.

[0260] Optionally, a second loss value is determined based on the sample spectrogram and the first predicted spectrogram. A third gradient information of the second loss value in this training with respect to the parameters of the speech synthesis model is determined. A fourth loss value is determined based on the sample spectrogram and the second predicted spectrogram. A fifth gradient information of the fourth loss value in this training with respect to the first speech feature is determined. The third gradient information, the fifth gradient information and the second gradient information are weighted and fused to obtain a third fused gradient information. Based on the third fused gradient information, the speech synthesis model, the feature extraction model, the phoneme clustering model and the phoneme mapping table are trained.

[0261] Optionally, the phoneme clustering model further includes a decoder, then the process of obtaining the second predicted spectrogram includes: decoding the second speech feature based on the decoder in the phoneme clustering model to obtain the second predicted spectrogram.

[0262] The decoder employs a linear transformation to decode the second speech feature, obtaining a second predicted spectrogram. This decoder can be an encoder of any form; for example, it could be a VQ-VAE.

[0263] It should be noted that the embodiments of this application are illustrated by taking the simultaneous training of the speech synthesis model, the feature extraction model, the phoneme clustering model and the phoneme mapping table as an example. In another embodiment, step 807 is not required. Instead, the speech synthesis model and the feature extraction model are trained only based on the sample spectrogram, the predicted spectrogram and the second gradient information.

[0264] In the solution provided in this application embodiment, the speech synthesis model, the feature extraction model, the phoneme clustering model, and the phoneme mapping table are trained based on the sample spectrogram, the predicted spectrogram, and the second gradient information. This ensures that while the feature extraction model and the speech synthesis model are less affected by noise, the accuracy of the phoneme clustering model and the phoneme mapping table are also improved. Consequently, the second feature extraction model, updated based on the phoneme clustering model and the phoneme mapping table, obtains a more accurate spectrogram from the speech synthesis model, further enhancing the accuracy of the speech synthesis model.

[0265] It should be noted that the embodiments described in this application are as described above. Figures 2 to 8 The embodiments shown are described separately, while in another embodiment, the above... Figures 2 to 8 The embodiments shown can be combined arbitrarily to... Figures 2 to 8 The illustrated embodiments are used as examples, such as Figure 9 As shown, the method includes:

[0266] 901. When a computer device has obtained the voice data collection permission for a target account, it acquires the voice data entered by the target account in the logged-in application and identifies the acquired voice data as the sample voice data corresponding to the target account.

[0267] 902. The computer equipment performs spectrum conversion on the sample speech data to obtain the sample spectrum diagram corresponding to the sample speech data.

[0268] 903. The computer device acquires the sample label corresponding to the sample speech data. The sample label indicates whether the sample speech data contains noise.

[0269] 904. The computer device obtains the first speech feature corresponding to the sample spectrogram based on the feature extraction model; and classifies the first speech feature based on the classification model to obtain the predicted label.

[0270] 905. The computer device obtains the gradient function of the target loss function with respect to the first parameter, the target loss function including a second parameter referring to the sample label and a third parameter referring to the predicted label, the third parameter including the first parameter, the first parameter referring to the first speech feature; based on the gradient function, the sample label, the predicted label and the first speech feature, the first gradient information of the loss value of this training with respect to the first speech feature is determined.

[0271] 906. Based on the first gradient information of the loss value of this training with respect to the first speech feature, the computer device determines the second gradient information that is inversely related to the first gradient information.

[0272] 907. The computer device, based on a phoneme clustering model, determines similar phoneme features for each sample phoneme feature in the first speech feature from multiple standard phoneme features contained in the phoneme mapping table; based on the phoneme clustering model, it constructs a second speech feature from the similar phoneme features of multiple sample phoneme features in the first speech feature. The second speech feature is then processed based on the phoneme clustering model to obtain a second predicted spectrogram.

[0273] 908. The computer device performs speech recognition on the sample speech data to obtain text information that matches the sample speech data, performs phoneme mapping on the text information, and obtains the phoneme sequence corresponding to the sample speech data.

[0274] 909. The computer device extracts features from the phoneme sequence based on the speech synthesis model to obtain the second phoneme feature information.

[0275] 910. Based on the speech synthesis model, the computer device adopts a self-attention mechanism to update the phoneme features in the second phoneme feature information, thereby obtaining the updated second phoneme feature information.

[0276] 911. The computer device, based on a speech synthesis model, expands the phoneme features corresponding to each phoneme in the updated second phoneme feature information according to the sample duration of each phoneme in the phoneme sequence, so as to obtain the first phoneme feature information, so that the number of phoneme features corresponding to each phoneme in the first phoneme feature information matches the corresponding sample duration.

[0277] 912. The computer device, based on the speech synthesis model, fuses the first phoneme feature information with the first speech feature to obtain a first fused feature; based on the speech synthesis model, it adopts a self-attention mechanism to update the first fused feature to obtain an updated first fused feature.

[0278] 913. The computer device, based on a speech synthesis model, fuses the updated first fusion feature with the second speech feature to obtain a second fusion feature; based on the speech synthesis model, it performs speech synthesis on the second fusion feature to obtain the first predicted spectrogram.

[0279] 914. Based on the speech synthesis model, the computer device performs duration prediction on the feature information of the second phoneme to obtain the predicted duration of each phoneme.

[0280] 915. The computer device trains the speech synthesis model, the feature extraction model, the classification model, the phoneme clustering model, and the phoneme mapping table based on the difference between the sample spectrogram and the first predicted spectrogram, the difference between the sample spectrogram and the second predicted spectrogram, the second gradient information, the sample duration and the predicted duration of each phoneme.

[0281] In the solution provided in this application embodiment, the difference between the sample spectrogram and the predicted spectrogram can reflect the accuracy of the speech synthesis model. A feature extraction model is used to obtain predicted labels indicating the likelihood of noise in the sample speech data. The loss value determined based on the sample label and the predicted label reflects the degree to which the speech features extracted by the feature extraction model are affected by noise. Furthermore, the loss value, with respect to the first gradient information of the speech features extracted by the feature extraction model, makes the speech features obtained by the trained feature extraction model more susceptible to noise, making it easier to classify whether the speech features obtained by the feature extraction model contain noise. Based on the sample spectrogram, the predicted spectrogram, and the loss value with respect to the first gradient information... The second gradient information is used to train the speech synthesis model and the feature extraction model, so that the speech features obtained by the feature extraction model tend to be unaffected by noise, and the speech features obtained by the feature extraction model are not easily classified as containing noise. Then, when the speech synthesis model generates a predicted spectrogram, it can learn speech features that tend to be unaffected by noise, so that the trained speech synthesis model has the function of generating spectrograms that tend to be noise-free. This ensures the quality of the spectrograms obtained by the speech synthesis model. Moreover, when training the speech synthesis model in this way, there is no need to specifically obtain high-quality and noise-free sample speech data, the requirements for sample speech data are low, and the applicability of this training method is improved.

[0282] Furthermore, based on the sample spectrogram, the predicted spectrogram, and the second gradient information, the speech synthesis model, the feature extraction model, the phoneme clustering model, and the phoneme mapping table are trained. This ensures that the feature extraction model and the speech synthesis model are less affected by noise, while also improving the accuracy of the phoneme clustering model and the phoneme mapping table. As a result, the second feature extraction model, updated based on the phoneme clustering model and the phoneme mapping table, obtains a more accurate spectrogram from the speech synthesis model, further enhancing the accuracy of the speech synthesis model.

[0283] Furthermore, by performing speech recognition and phoneme mapping on the sample speech data, the phoneme sequence corresponding to the sample speech data is obtained. Since the phoneme sequence obtained by this method may contain errors, a second speech feature is obtained based on the feature extraction model, combined with a phoneme clustering model and a phoneme mapping table, to obtain the predicted spectrogram. The speech synthesis model, the feature extraction model, the classification model, the phoneme clustering model, and the phoneme mapping table are then trained to reduce the impact of errors in the phoneme sequence obtained by the speech recognition and phoneme mapping methods. As a result, the spectrogram generated by the trained speech synthesis model is accurate enough and noise-free, and the speech data converted from the spectrogram generated by the trained speech synthesis model has the same timbre as the user to whom the target account belongs.

[0284] In the above Figure 9 Based on the embodiments shown, this application also provides a process for generating predicted spectrograms based on multiple models, such as... Figure 10 As shown, the multiple models include a speech synthesis model 1001, a feature extraction model 1002, a classification model 1003, and a phoneme clustering model 1004. The speech synthesis model 1001 includes a phoneme coding layer, a self-attention coding layer, a duration prediction layer, a self-attention decoding layer, and a linear layer; the classification model 1003 includes a transform sub-model and a classification sub-model, taking the transform sub-model as a linear layer and the classification sub-model 1003 as a classifier as an example; the phoneme clustering model 1004 includes an encoder, a phoneme mapping table, and a decoder.

[0285] In step 904 above, the process of obtaining the predicted label includes: performing feature transformation on the first speech feature based on the linear layer in the classification model to obtain the third speech feature; and classifying the third speech feature based on the classifier in the classification model to obtain the predicted label.

[0286] Step 907 above includes: encoding the first speech feature based on the encoder in the phoneme clustering model to obtain the fourth speech feature; determining the similar phoneme features of each sample phoneme feature in the fourth speech feature from multiple standard phoneme features contained in the phoneme mapping table based on the phoneme clustering model, and constructing the similar phoneme features of multiple sample phoneme features in the fourth speech feature as the second speech feature; and decoding the second speech feature based on the decoder in the phoneme clustering model to obtain the second predicted spectrogram.

[0287] Step 909 is implemented based on the phoneme coding layer in the speech synthesis model; step 910 is implemented based on the self-attention coding layer in the speech synthesis model; step 911 is implemented based on the duration prediction layer in the speech synthesis model; step 912 is implemented based on the self-attention decoding layer in the speech synthesis model; step 913 is implemented based on the linear layer in the speech synthesis model; and step 914 is implemented based on the duration prediction layer in the speech synthesis model. The self-attention coding layer can be any network model, for example, a Transformer. The self-attention decoding layer can also be any network model, for example, a Transformer.

[0288] It should be noted that, in the above Figures 2 to 10Based on the illustrated embodiment, after training the speech synthesis model, it is also possible to acquire speech data based on the trained speech synthesis model. That is, the process of acquiring speech data includes: acquiring target text information; performing phoneme mapping on the target text information to obtain the target phoneme sequence corresponding to the target text information; processing the target phoneme sequence based on the target speech synthesis model to obtain the target spectrogram; and performing speech conversion on the target spectrogram to obtain the target speech data corresponding to the target text information.

[0289] The target text information is the text information to be used to generate voice data. The target phoneme sequence includes at least one phoneme, and the pronunciation of the at least one phoneme matches the pronunciation of the target text information. The target voice data is voice data with the same timbre as the user to whom the target account belongs.

[0290] In one possible implementation, the target spectrogram is converted into speech data based on a vocoder to obtain the target speech data corresponding to the target text information. The vocoder is used to convert the spectrogram into speech data.

[0291] In one possible implementation, the process of obtaining the target spectrogram based on the target speech synthesis model includes: a computer device extracting features from the target phoneme sequence based on the speech synthesis model to obtain third phoneme feature information; updating the phoneme features in the third phoneme feature information using a self-attention mechanism based on the speech synthesis model to obtain updated third phoneme feature information; predicting the duration of the third phoneme feature information based on the speech synthesis model to obtain the predicted duration of each phoneme in the target phoneme sequence; expanding the phoneme features corresponding to each phoneme in the updated third phoneme feature information according to the predicted duration of each phoneme in the target phoneme sequence based on the speech synthesis model to obtain fourth phoneme feature information, so that the number of phoneme features corresponding to each phoneme in the fourth phoneme feature information matches the corresponding predicted duration; updating the fourth phoneme feature information using a self-attention mechanism based on the speech synthesis model to obtain updated fourth fusion features; and performing speech synthesis on the updated fourth fusion features based on the speech synthesis model to obtain the target spectrogram.

[0292] In one possible implementation, this application can also combine any emotion type to obtain target speech data. That is, the process of obtaining target speech data includes: obtaining the target emotion type; processing the target phoneme speech sequence based on the target speech synthesis model and the target emotion type to obtain a target spectrogram; and performing speech conversion on the target spectrogram to obtain target speech data corresponding to the target text information and belonging to the target emotion type.

[0293] Optionally, the target sentiment type is specified by the user or obtained by sentiment recognition of the target text information.

[0294] Optionally, the process of obtaining the target spectrogram based on the target speech synthesis model and the target emotion type includes: obtaining the emotion features corresponding to the target emotion type; then, following the above process of obtaining the target spectrogram based on the target speech synthesis model, fusing each phoneme feature in the obtained third phoneme feature information with the emotion feature; then, based on the speech synthesis model, using a self-attention mechanism, updating the phoneme features in the fused phoneme feature information to obtain the updated third phoneme feature information; or, after obtaining the updated third phoneme feature information, fusing each phoneme feature in the updated third phoneme feature information with the emotion feature; then, based on the speech synthesis model, performing duration prediction on the fused phoneme feature information to obtain the predicted duration of each phoneme in the target phoneme sequence; and based on the speech synthesis model, expanding the phoneme features corresponding to each phoneme in the fused phoneme feature information according to the predicted duration of each phoneme in the target phoneme sequence to obtain the fourth phoneme feature information.

[0295] Figure 11 This application provides a method for acquiring voice data, such as... Figure 11 As shown, this method is executed by a computer device. Taking the computer device as an example as the terminal, the method includes:

[0296] 1101. The terminal displays the model training interface, which shows the first prompt message.

[0297] The first prompt message is used to prompt authorization to collect the voice data input by the target account in the current application, and the collected voice data is used to train the target voice synthesis model corresponding to the target account. The voice data obtained by converting the spectrogram generated based on the target voice synthesis model is the same as the timbre of the user to which the target account belongs.

[0298] In one possible implementation, step 1101 includes: the terminal displays a model training interface, which has introductory information and training options, and in response to a triggering operation on the training option, displays the first prompt information in the model training interface.

[0299] The introductory information describes the purpose and usage of the speech synthesis model. The training option initiates the process of authorizing the collection of speech data. The training option can be any form of option, such as a button.

[0300] In one possible implementation, the terminal displays the model training interface based on the current application.

[0301] The current application can be any application, such as a voice application or an instant messaging application.

[0302] 1102. The terminal responds to the confirmation operation of the first prompt information and obtains the voice data collection permission of the target account.

[0303] Specifically, the voice data collection permission indicator allows the use of voice data input based on the target account in the current application to train the target speech synthesis model. Since the first prompt message is used to authorize the collection of voice data input by the target account in the current application, the terminal detects a confirmation operation on the first prompt message, indicating that the user agrees to the authorization, and thus obtains the voice data collection permission.

[0304] In one possible implementation, the model training interface also displays a confirmation option. In this case, step 1102 includes: the terminal, in response to a triggering operation on the confirmation option, acquiring voice data collection permissions for the target account. The confirmation option is used to confirm the first prompt information. When the terminal detects the user's triggering operation on the confirmation option, it indicates that the user agrees to the authorization, and thus acquires voice data collection permissions for the target account.

[0305] Optionally, the model training interface also displays a denial option. When the terminal responds to the denial option, indicating that the user denies authorization, it will not obtain the voice data collection permission for the target account.

[0306] In one possible implementation, step 1102 includes: in response to a confirmation operation on the first prompt information, displaying a second prompt information and acquiring input voice data; if the acquired voice data matches the preset text included in the second prompt information, acquiring voice data collection permissions for the target account.

[0307] The second prompt includes preset text, which prompts the user to input voice data according to the preset text. The preset text can contain any content; for example, the preset text could be "Who am I?"

[0308] The solution provided in this application adopts a voice authorization method. The user inputs voice data according to the content prompted by the second prompt information. Only when the input voice data matches the preset text contained in the prompt information is the voice data collection permission of the target account obtained. This realizes a voice authorization method and can ensure that the terminal can input voice data normally, thereby ensuring that the voice data based on the account input can be collected subsequently.

[0309] Optionally, after acquiring the input voice data, speech recognition is performed on the voice data to obtain text information. The recognized text information is compared with the predicted text. If the recognized text information is the same as the predicted text, it is determined that the acquired voice data matches the preset text included in the second prompt information. If the recognized text information is different from the predicted text, it is determined that the acquired voice data does not match the preset text included in the second prompt information.

[0310] It should be noted that this embodiment of the application takes the input of voice data once as an example. When the acquired voice data matches the preset text included in the second prompt information, the voice data collection permission of the target account is obtained. In another embodiment, the above steps can be repeated multiple times. That is, when the acquired voice data matches the preset text included in the second prompt information, a third prompt information is displayed. The third prompt information includes other preset texts. The input voice data is acquired again. When the acquired voice data matches the third prompt information including other preset texts, the voice data collection permission of the target account is obtained.

[0311] 1103. When the terminal obtains the voice data collection permission of the target account, it obtains the voice data input by the target account in the current application.

[0312] In this embodiment of the application, the voice data collection permission is the permission to collect voice data input based on the target account in the current application. Therefore, if the voice data collection permission of the target account is obtained, and voice data is input based on the target account in the current application, the voice data is obtained so that it can be used as sample voice data for training the target speech synthesis model corresponding to the target account.

[0313] In one possible implementation, the application server is used to provide services for the current application, and the training server is used to train the speech synthesis model. Step 1103 includes: when the terminal obtains the speech data collection permission of the target account, it obtains the speech data input by the target account in the current application, sends the speech data to the application server, the application server sends the speech data to the training server, and the training server stores the speech data corresponding to the target account so that the training server can subsequently train the target speech synthesis model for the target account based on the speech data stored corresponding to the target account.

[0314] Optionally, following steps 1101-1103 above, the terminal obtains voice data collection permissions for the target account in multiple applications. The terminal collects voice data input by the target account in any application and sends the collected voice data to the application server that provides services for the application. The application server, having obtained voice data collection permissions for the target account, sends the voice data received by the application server to the training server. The training server then stores the voice data corresponding to the target account.

[0315] For example, if a user authorizes voice data collection in four applications based on a target account, meaning that the application servers of all four applications have obtained voice data collection permissions for the target account, then when the user inputs voice data in any of the four applications based on the target account, the application server of that application interacts with the training server to send the input voice data to the training server for storage.

[0316] Optionally, after the training server obtains the voice data corresponding to the target account, it proceeds according to the above... Figures 2 to 10 The example shown trains the target speech synthesis model corresponding to the target account.

[0317] In the solution provided in this application embodiment, the voice data input based on the target account in the current application is authorized to be collected through the displayed model training interface, so that the collected voice data model can be used to train a voice synthesis model for the target account in the future. This realizes a solution for authorized collection of voice data for training a voice synthesis model, which is convenient for users to operate and allows users to know the purpose of the authorized collection of voice data, thereby improving the user experience.

[0318] Based on the above Figure 11 As shown in the embodiments, this application also provides a flowchart for user confirmation authorization, such as... Figure 12 and Figure 13As shown, the terminal displays a model training interface. The user clicks the "Train My Speech Synthesis Model" button 1301 in the model training interface to initiate a training request. The first prompt message 1302 is displayed on the model training interface, indicating the authorization process and the purpose of the acquired speech data. The user clicks the "Agree" button 1303, and a fourth prompt message 1304 is displayed. This fourth prompt message 1304 indicates that the speech authorization stage is about to begin and specifies the number of times speech data needs to be recorded for the authorization. The user clicks the "Next" button 1305, and a second prompt message 1306 is displayed on the model training interface. This second prompt message 1306 includes preset text, which includes "XXXX". If the user selects "XX", the terminal will input voice data according to the content of the preset text. The terminal will collect the user's voice data and compare it with the preset text. If the collected voice data matches the preset text, the terminal will display the next prompt message, which includes the preset text. The user will then input the next voice data according to the method described above based on the second prompt message 1306. If the number of voice authorization attempts reaches the set number and the collected voice data matches the preset text displayed during each voice authorization, the terminal will display the authorization completion prompt message 1307. The authorization completion prompt message 1307 indicates that a model will be trained for the user and that the user can authorize the voice synthesis model to be used by multiple applications.

[0319] It should be noted that, in the above Figure 11 Based on the illustrated embodiment, the training status of the speech synthesis model can be viewed in the model display interface, and speech can be generated and listened to based on the currently trained speech synthesis model.

[0320] In one possible implementation, the target speech synthesis model's status information is displayed in the model display interface, indicating the training progress of the target speech synthesis model across multiple emotion types.

[0321] Each emotion type represents a specific emotion; for example, these emotion types could be happiness, excitement, calmness, and sadness. The training progress of the speech synthesis model on any given emotion type represents the number of times the model has been trained using speech data belonging to that emotion type, or the number of speech data entries belonging to that emotion type used in training the model. This reflects the quality of the speech data generated by the model for that emotion type. For any given emotion type, a greater training progress indicates higher quality speech data, while a smaller training progress indicates lower quality. Furthermore, if the training progress on any emotion type is incomplete, it also indicates a shortage of speech data belonging to that emotion type used in training the model, suggesting that more speech data of that emotion type is needed to improve the model's training progress. Users can then input speech data belonging to that emotion type into authorized applications via their terminals to track the training progress of the speech synthesis model on any emotion type. This will allow for the collection of more speech data belonging to that emotion type to train the speech synthesis model, thereby accelerating the training progress of the speech synthesis model on any emotion type.

[0322] The status information can be represented in any form. For example, the status information can be represented as a progress bar, which includes multiple progress bars corresponding to different emotion types. The progress in each emotion type's progress bar indicates the training progress of the speech synthesis model for that emotion type. Alternatively, the status information can be represented as a polygonal graphic, where one corner of the polygonal graphic represents an emotion type, and the patterns filled in the polygonal graphic reflect the training progress of the speech synthesis model for each emotion type.

[0323] Optionally, the model display interface also displays the first score of the target speech synthesis model. The first score indicates the quality of the speech data generated based on the target speech synthesis model. The higher the first score, the higher the quality of the speech data generated based on the target speech synthesis model.

[0324] Optionally, a second score for the target speech synthesis model across multiple emotion types is determined, and the second scores across multiple emotion types are fused to obtain the first score. The second score indicates the quality of the speech data generated based on the target speech synthesis model for the corresponding emotion type.

[0325] In one possible implementation, the model display interface includes a text input area and an emotion selection area; the text input area is used to input the text information of the speech data to be generated, and the emotion selection area is used to select the emotion type of the speech data to be generated. The process of synthesizing and listening to speech data through the model display interface includes: responding to a speech synthesis instruction in the model display interface, obtaining the text information input in the text input area, and obtaining the selected emotion type in the emotion selection area; generating speech data that matches the text information and belongs to the emotion type based on the target speech synthesis model, the emotion type, and the text information; and playing the speech data.

[0326] In this embodiment of the application, the training status of the speech synthesis model can be viewed in the model display interface, and the speech data can be generated for listening using the currently trained speech synthesis model in the model display interface, so that the user can know the training status of the currently trained speech synthesis model.

[0327] Each emotion type indicates a specific emotion, and the speech synthesis instruction instructs the generation of speech data based on the text information entered in the text input area and the emotion type selected in the emotion selection area. Optionally, the model's interface includes a target option, and triggering this target option is equivalent to receiving a speech synthesis instruction.

[0328] Optionally, the emotion selection area includes multiple emotion types, and the emotion selection area can be represented in any form. For example, the emotion selection area includes options corresponding to multiple emotion types; when a user clicks on an option, the emotion type corresponding to that option is the selected emotion type. As another example, the emotion selection area can be represented as a circular area, with different positions within the area corresponding to different emotion types; when a user selects any position within the area, the emotion type corresponding to the selected position is the selected emotion type.

[0329] Optionally, the model display interface also shows the total duration of the speech data obtained from multiple applications. In this embodiment, by viewing the model display interface, the user can obtain the total duration of the speech data obtained from various applications, thus understanding the specific details of the speech data used in training the speech synthesis model.

[0330] like Figure 14As shown, the model display interface displays the total duration of voice data obtained from multiple applications and an "Experience" button 1401. When the user clicks the "Experience" button 1401, a text input area 1402, an emotion selection area 1403, and a "Listen" button 1404 are displayed on the model display interface. The user can input any text information in the text input area 1402 and select any emotion in the emotion selection area 1403, such as selecting "happy". Then, when the user clicks the "Listen" button 1404, voice data with the emotion of "happy" that matches the text information is generated and played, thus realizing the voice data listening solution.

[0331] It should be noted that, in the embodiments of this application, after generating speech data in response to the speech synthesis command in the model display interface, the speech data can also be shared or applied to the constructed virtual character.

[0332] For example, the model's interface also displays a share button. When a user clicks the share button on their terminal, a friend list is displayed. If any friend's account is selected in the friend list, the terminal logged into that user's account will send the voice data to the terminal logged into that friend's account, so that the terminal can play the voice data. This achieves the solution of using the voice synthesis model to speak for the user and communicate with friends via voice.

[0333] For example, the model's display interface also shows a download button. By clicking the download button and selecting a storage location, the user can store the voice data in the target storage location. This target storage location is used to store the voice data configured for the virtual character created by the user in the target application. In this way, the user's speech can be simulated using the virtual character and the voice data.

[0334] It should be noted that the above Figures 2 to 10 The embodiments shown are compatible with the above-described embodiments. Figure 11 The illustrated embodiments are combined, that is, first according to the above... Figure 11 The embodiment shown authorizes the acquisition of voice data from the target account, and then, first follows the above... Figures 2 to 10 The illustrated embodiment uses the voice data of the target account to train the speech synthesis model.

[0335] In one possible implementation, according to the above... Figure 11 The illustrated embodiment obtains the target account's data collection permissions in the target application, and obtains the voice data input by the target account in the target application. Based on the obtained voice data, it is processed according to the above... Figures 2 to 10The illustrated embodiment trains a target speech synthesis model corresponding to the target account. Afterwards, the user can set the application scenarios for the trained target speech synthesis model within the target application. For example, the user can authorize the target application to use it. Subsequently, when sending voice data or making voice calls based on the target application, the user can input text information. Based on the target speech synthesis model and the text information, a spectrogram is generated, and then the spectrogram is converted into voice data for transmission, thus realizing the solution of the target speech synthesis model speaking on behalf of the user. As another example, the user can authorize the target speech synthesis model to be used by other applications.

[0336] The speech model training method and speech data acquisition method provided in this application can be applied to various scenarios. The trained speech synthesis model can replace the user's speech. When the user does not want to speak, they only need to input text information. Based on the text information and the speech synthesis model, speech data with the same timbre as the user is generated. The generated speech data is used as the user's input speech data. In this way, the speech synthesis model can replace the user's speech, thereby improving the user experience.

[0337] For example, in a phone call scenario, the user inputs text information into the terminal, which then calls the voice model and combines it with the text information to generate voice data with the same timbre as the user. The voice data is then sent to the other party. During this process, the user does not need to speak, thus enabling telephone communication with others.

[0338] For example, in the case of recording audiobooks, users can use their terminal to call their target speech synthesis model to convert the book text into a spectrogram, and then convert the spectrogram into speech data with the same timbre as the user. In other words, the resulting speech data is an audiobook recorded with the user's voice, eliminating the need for manual recording of audiobooks and thus saving labor costs.

[0339] For example, in a live streaming scenario, the host calls the target speech synthesis model through the terminal. The host only needs to input text information into the terminal. Based on the target speech synthesis model and the input text information, the terminal generates speech data with the same timbre as the host and publishes the speech data in the live streaming room, realizing the solution of the host conducting live streaming based on the target speech synthesis model.

[0340] For example, after obtaining their target speech synthesis model, a user can license that model to others, such as a navigation application, so that the application can generate navigation voices with the same timbre as the user based on the model. Another example is if user 1 licenses their target speech synthesis model to user 2, then user 2 can generate speech data with the same timbre as user 1 based on the model, thus simulating a voice chat between user 2 and user 1.

[0341] For example, the method provided in the embodiments of this application can utilize the user's voice data from various age groups to train target speech synthesis models corresponding to each age group. This allows the user to generate voice data with the same timbre as the user at any given age based on the target speech synthesis model corresponding to that age group. For instance, if a user generates multiple speech synthesis models using voice data from their childhood, adolescence, youth, middle age, and old age, then the speech synthesis model corresponding to childhood can generate voice data with the same timbre as the user during childhood. Furthermore, this speech synthesis model can serve as the user's digital resource, which the user can use themselves or authorize others to use.

[0342] For example, in a gaming scenario, when a user runs a game application on a terminal, the user inputs text information into the game application. The terminal then uses the game application to call the user's target speech synthesis model to generate speech data with the same timbre as the user. The generated speech data is then sent to the terminals of other users participating in the game, so that their terminals can play the speech data. This enables voice chat with other users participating in the game using a speech synthesis model.

[0343] For example, in a virtual character intelligent question-and-answer scenario, the first user can have a voice conversation with the virtual character. While the virtual character is speaking, the second user's voice synthesis model is used to generate voice data with the same timbre as the second user and play it. This realizes the solution of the first user having a voice conversation with the virtual character, and the timbre of the voice data emitted by the virtual character is the same as that of the second user, simulating the scenario of the first user having a voice conversation with the second user, thereby improving the user experience.

[0344] Figure 15 This is a schematic diagram of the structure of a speech synthesis model training device provided in an embodiment of this application, as shown below. Figure 15 As shown, the device includes:

[0345] The acquisition module 1501 is used to acquire the sample spectrogram and sample label corresponding to the sample speech data. The sample label indicates whether the sample speech data contains noise.

[0346] The classification module 1502 is used to obtain the first speech feature corresponding to the sample spectrogram based on the feature extraction model; classify the first speech feature to obtain a prediction label, and the prediction label indicates the possibility that the sample speech data contains noise.

[0347] The determination module 1503 is used to determine the second gradient information that is inverse to the first gradient information based on the first gradient information of the loss value of the current training with respect to the first speech feature. The loss value is determined based on the sample label and the predicted label.

[0348] The generation module 1504 is used to generate a predicted spectrogram based on the speech synthesis model and the first speech features;

[0349] Training module 1505 is used to train the speech synthesis model and feature extraction model based on the sample spectrogram, the predicted spectrogram, and the second gradient information.

[0350] In the scheme provided in this application embodiment, adversarial training is performed by combining a feature extraction model and a speech synthesis model. The speech features obtained by the feature extraction model are used in the speech synthesis model. This process involves training not only based on the sample spectrogram and predicted spectrogram corresponding to the sample speech data, but also considering that the loss value determined based on the sample labels and predicted labels can reflect the accuracy of whether the speech features extracted by the feature extraction model contain noise. If the feature extraction model is trained based on the first gradient information of the speech features extracted by the feature extraction model with respect to this loss value, the speech features obtained by the feature extraction model can include noise-related features, making it easier to distinguish whether noise is present. Therefore, a method of inverting the gradient information is adopted, and training is performed based on the second gradient information, which is the inverse of the first gradient information. This makes the speech features obtained by the feature extraction model tend to not contain noise-related features, that is, the speech features obtained by the feature extraction model are not affected by noise. The speech features obtained by the feature extraction model are then used to train the speech synthesis model, enabling the trained speech synthesis model to generate noise-free spectrograms. This training method does not require high-quality and noise-free sample speech data, has low requirements for sample speech data, and has a wider range of applications.

[0351] In one possible implementation, the acquisition module 1501 is further configured to acquire the gradient function of the target loss function with respect to the first parameter. The target loss function includes a second parameter representing the sample label and a third parameter representing the predicted label. The third parameter includes the first parameter, which represents the first speech feature. The gradient function represents the relationship between the first gradient information and the sample label, the predicted label, and the first speech feature.

[0352] The determination module 1503 is also used to determine the first gradient information based on the gradient function, sample label, predicted label and first speech features.

[0353] In another possible implementation, the first speech feature includes multiple sample phoneme features; the generation module 1504 is used to determine the similar phoneme features of each sample phoneme feature from multiple standard phoneme features contained in the phoneme mapping table; the similar phoneme features of multiple sample phoneme features constitute the second speech feature; and a predicted spectrogram is generated based on the speech synthesis model, the first speech feature and the second speech feature.

[0354] In another possible implementation, the phoneme mapping table is included in the phoneme clustering model, and the step of obtaining the second speech feature based on the first speech feature is implemented based on the phoneme clustering model; the training module 1505 is used to train the speech synthesis model, the feature extraction model, the phoneme clustering model and the phoneme mapping table based on the sample spectrogram, the predicted spectrogram and the second gradient information.

[0355] In another possible implementation, the generation module 1504 is used to process the phoneme sequence and first speech features corresponding to the sample speech data based on the speech synthesis model to obtain a predicted spectrogram. The phoneme sequence includes at least one phoneme, and the pronunciation of at least one phoneme matches the sound in the sample speech data.

[0356] In another possible implementation, the generation module 1504 is used to obtain the first phoneme feature information corresponding to the phoneme sequence; fuse the first phoneme feature information with the first speech feature to obtain the fused feature; and perform speech synthesis on the fused feature to obtain the predicted spectrogram.

[0357] In another possible implementation, the generation module 1504 is used to extract features from the phoneme sequence to obtain second phoneme feature information, which includes phoneme features corresponding to each phoneme in the phoneme sequence; according to the sample duration of each phoneme, the phoneme features corresponding to each phoneme in the second phoneme feature information are expanded to obtain first phoneme feature information, so that the number of phoneme features corresponding to each phoneme in the first phoneme feature information matches the corresponding sample duration, and the sample duration indicates the duration of the phoneme's pronunciation in the sample speech data.

[0358] In another possible implementation, such as Figure 16 As shown, the device also includes:

[0359] The prediction module 1506 is used to predict the duration of the second phoneme feature information based on the speech synthesis model, so as to obtain the predicted duration of each phoneme.

[0360] Training module 1505 is used to train the speech synthesis model and feature extraction model based on the sample spectrogram, the predicted spectrogram, the second gradient information, the sample duration and the predicted duration of each phoneme.

[0361] In another possible implementation, such as Figure 16 As shown, the device also includes:

[0362] The recognition module 1507 is used to perform emotion recognition on the sample speech data to obtain the emotion type of the sample speech data;

[0363] The acquisition module 1501 is also used to acquire the emotional features corresponding to the emotional type;

[0364] The generation module 1504 is used to process the phoneme sequence, emotion features and first speech features based on the speech synthesis model to obtain the predicted spectrogram.

[0365] In another possible implementation, such as Figure 16 As shown, the device also includes:

[0366] The acquisition module 1501 is also used to acquire target text information;

[0367] The mapping module 1508 is used to perform phoneme mapping on the target text information to obtain the target phoneme sequence corresponding to the target text information.

[0368] The processing module 1509 is used to process the target phoneme sequence based on the target speech synthesis model to obtain the target spectrogram;

[0369] The conversion module 1510 is used to convert the target spectrogram into speech to obtain the target speech data corresponding to the target text information.

[0370] In another possible implementation, the classification module 1502 is used to classify the first speech features based on the classification model to obtain the predicted label;

[0371] Training module 1505 is used to train speech synthesis model, feature extraction model and classification model based on sample spectrogram, predicted spectrogram and second gradient information.

[0372] In another possible implementation, the acquisition module 1501 is used to acquire voice data input by the target account in the logged-in application, provided that the voice data acquisition permission of the target account has been acquired. The voice data acquisition permission indicates that the voice data input by the target account in the application is allowed to be used to train the target speech synthesis model corresponding to the target account. The acquired voice data is determined as sample voice data. The sample voice data is subjected to spectrum conversion to obtain a sample spectrum diagram.

[0373] In another possible implementation, the acquisition module 1501 is also used to acquire the sample spectrogram corresponding to the sample voice data based on input from multiple accounts;

[0374] Training module 1505 is also used to train a speech synthesis model based on the sample spectrograms corresponding to multiple accounts.

[0375] It should be noted that the speech synthesis model training device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the speech synthesis model training device and the speech synthesis model training method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0376] Figure 17 This is a schematic diagram of the structure of a voice data acquisition device provided in an embodiment of this application, as shown below. Figure 17 As shown, the device includes:

[0377] Display module 1701 is used to display the model training interface. The model training interface displays a first prompt message. The first prompt message is used to prompt authorization to collect the voice data input by the target account in the current application, and the collected voice data is used to train the target speech synthesis model corresponding to the target account.

[0378] The acquisition module 1702 is used to acquire the voice data acquisition permission of the target account in response to the confirmation operation of the first prompt information. The voice data acquisition permission indicates that the voice data input based on the target account in the current application is allowed to be used to train the target speech synthesis model.

[0379] The acquisition module 1702 is also used to acquire the voice data input by the target account in the current application, provided that the target account has been granted permission to collect voice data.

[0380] In the solution provided in this application embodiment, the voice data input based on the target account in the current application is authorized to be collected through the displayed model training interface, so that the collected voice data model can be used to train a voice synthesis model for the target account in the future. This realizes a solution for authorized collection of voice data for training a voice synthesis model, which is convenient for users to operate and allows users to know the purpose of the authorized collection of voice data, thereby improving the user experience.

[0381] In one possible implementation, the acquisition module 1702 is used to display a second prompt message in response to a confirmation operation of the first prompt message. The second prompt message includes preset text and is used to prompt the user to input voice data according to the preset text. The module acquires the input voice data and, if the acquired voice data matches the preset text, acquires the voice data collection permission of the target account.

[0382] In another possible implementation, the display module 1701 is also used to display the status information of the target speech synthesis model in the model display interface, the status information indicating the training progress of the target speech synthesis model on multiple emotion types.

[0383] In another possible implementation, the model display interface includes a text input area and a sentiment selection area; such as Figure 18 As shown, the device also includes:

[0384] The acquisition module 1702 is also used to, in response to the speech synthesis instruction in the model display interface, acquire the text information entered in the text input area and acquire the emotion type selected in the emotion selection area;

[0385] The generation module 1703 is used to generate speech data that matches the text information and belongs to the emotion type based on the target speech synthesis model, emotion type and text information;

[0386] The playback module 1704 is used to play audio data.

[0387] It should be noted that the voice data acquisition device provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the voice data acquisition device and the voice data acquisition method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0388] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one computer program, which is loaded and executed by the processor to implement the operations performed by the speech synthesis model training method or speech data acquisition method of the above embodiments.

[0389] Optionally, the computer device is provided as a terminal. Figure 19A structural block diagram of a terminal 1900 provided in an exemplary embodiment of this application is shown. The terminal 1900 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, or in-vehicle terminal, etc. The terminal 1900 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0390] Terminal 1900 includes a processor 1901 and a memory 1902.

[0391] Processor 1901 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1901 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1901 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1901 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1901 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0392] The memory 1902 may include one or more computer-readable storage media, which may be non-transitory. The memory 1902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1902 are used to store at least one computer program, which is executed by the processor 1901 to implement the speech synthesis model training method or speech data acquisition method provided in the method embodiments of this application.

[0393] In some embodiments, the terminal 1900 may also optionally include a peripheral device interface 1903 and at least one peripheral device. The processor 1901, memory 1902, and peripheral device interface 1903 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1903 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1904, a display screen 1905, a camera assembly 1906, an audio circuit 1907, and a power supply 1908.

[0394] Peripheral interface 1903 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1901 and memory 1902. In some embodiments, processor 1901, memory 1902 and peripheral interface 1903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1901, memory 1902 and peripheral interface 1903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0395] The radio frequency (RF) circuit 1904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1904 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1904 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1904 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0396] Display screen 1905 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1905 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1901 for processing. In this case, display screen 1905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1905, disposed on the front panel of terminal 1900; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1900 or in a folded design; in still other embodiments, display screen 1905 may be a flexible display screen, disposed on a curved or folded surface of terminal 1900. Furthermore, display screen 1905 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1905 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0397] The camera assembly 1906 is used to acquire images or videos. Optionally, the camera assembly 1906 includes a front-facing camera and a rear-facing camera. The front-facing camera is disposed on the front panel of the terminal, and the rear-facing camera is disposed on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1906 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cool light flash, which can be used for light compensation at different color temperatures.

[0398] The audio circuit 1907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting them into electrical signals that are input to the processor 1901 for processing, or to the radio frequency circuit 1904 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1900. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1901 or the radio frequency circuit 1904 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1907 may also include a headphone jack.

[0399] Power supply 1908 is used to power the various components in terminal 1900. Power supply 1908 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1908 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0400] Those skilled in the art will understand that Figure 19 The structure shown does not constitute a limitation on terminal 1900 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0401] Optionally, the computer equipment is provided as a server. Figure 20 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 2000 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 2001 and one or more memories 2002. The memories 2002 store at least one computer program, which is loaded and executed by the processor 2001 to implement the methods provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated upon here.

[0402] This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the operations performed by the speech synthesis model training method or speech data acquisition method of the above embodiments.

[0403] This application also provides a computer program product, including a computer program that, when executed by a processor, performs the operations performed by the speech synthesis model training method or the speech data acquisition method as described in the above embodiments.

[0404] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0405] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present application should be included within the protection scope of the present application.

Claims

1. A method for training a speech synthesis model, characterized in that, The method includes: Obtain the sample spectrogram and sample label corresponding to the sample speech data, wherein the sample label indicates whether the sample speech data contains noise; Based on the feature extraction model, the first speech feature corresponding to the sample spectrogram is obtained; the first speech feature is classified to obtain a prediction label, and the prediction label indicates the possibility that the sample speech data contains noise. Based on the first gradient information of the loss value of this training with respect to the first speech feature, a second gradient information that is inversely related to the first gradient information is determined, wherein the loss value is determined based on the sample label and the predicted label; Based on the phoneme clustering model, similar phoneme features of each sample phoneme feature within the first speech feature are determined from multiple standard phoneme features contained in the phoneme mapping table; similar phoneme features of all sample phoneme features are used to construct the second speech feature; the second speech feature is processed to obtain the second predicted spectrogram. Based on the speech synthesis model, the phoneme sequence corresponding to the sample speech data and the first speech feature are processed to obtain a first predicted spectrogram. The phoneme sequence includes at least one phoneme, and the pronunciation of the at least one phoneme matches the sound in the sample speech data. Based on the differences between the sample spectrogram and the first predicted spectrogram, the differences between the sample spectrogram and the second predicted spectrogram, and the second gradient information, the speech synthesis model, the feature extraction model, the phoneme clustering model, and the phoneme mapping table are trained.

2. The method according to claim 1, characterized in that, Before determining the second gradient information, which is the inverse of the first gradient information, based on the loss value of this training with respect to the first speech feature, the method further includes: Obtain the gradient function of the target loss function with respect to the first parameter. The target loss function includes a second parameter representing the sample label and a third parameter representing the predicted label. The third parameter includes the first parameter, and the first parameter represents the first speech feature. The gradient function represents the relationship between the first gradient information and the sample label, the predicted label, and the first speech feature. The first gradient information is determined based on the gradient function, the sample label, the predicted label, and the first speech feature.

3. The method according to claim 1, characterized in that, The step of processing the phoneme sequence corresponding to the sample speech data and the first speech feature to obtain the first predicted spectrogram includes: Obtain the first phoneme feature information corresponding to the phoneme sequence; The first phoneme feature information is fused with the first speech feature to obtain the fused feature; The fused features are used for speech synthesis to obtain the first predicted spectrogram.

4. The method according to claim 3, characterized in that, The step of obtaining the first phoneme feature information corresponding to the phoneme sequence includes: Feature extraction is performed on the phoneme sequence to obtain second phoneme feature information, which includes the phoneme features corresponding to each phoneme in the phoneme sequence. According to the sample duration of each phoneme, the phoneme features corresponding to each phoneme in the second phoneme feature information are expanded to obtain the first phoneme feature information, so that the number of phoneme features corresponding to each phoneme in the first phoneme feature information matches the corresponding sample duration, wherein the sample duration indicates the duration of the pronunciation of the phoneme in the sample speech data.

5. The method according to claim 1, characterized in that, The method further includes: Emotion recognition is performed on the sample speech data to obtain the emotion type to which the sample speech data belongs; Obtain the emotional features corresponding to the emotional type; The process of processing the phoneme sequence corresponding to the sample speech data and the first speech feature based on the speech synthesis model to obtain the first predicted spectrogram includes: Based on the speech synthesis model, the phoneme sequence, the emotion feature, and the first speech feature are processed to obtain the first predicted spectrogram.

6. The method according to any one of claims 3-5, characterized in that, The method further includes: Obtain target text information; Phoneme mapping is performed on the target text information to obtain the target phoneme sequence corresponding to the target text information; Based on the speech synthesis model, the target phoneme sequence is processed to obtain the target spectrogram; The target spectrogram is converted into speech to obtain the target speech data corresponding to the target text information.

7. The method according to any one of claims 1-5, characterized in that, The acquisition of the sample spectrogram corresponding to the sample speech data includes: If the voice data collection permission of the target account has been obtained, the voice data input by the target account in the logged-in application is obtained, and the voice data collection permission indicates that the voice data input by the target account in the application is allowed to be used to train the target speech synthesis model corresponding to the target account. The acquired voice data is identified as the sample voice data; The sample speech data is subjected to spectral conversion to obtain the sample spectrogram.

8. A method for acquiring voice data, characterized in that, The method includes: The model training interface is displayed, and the model training interface displays a first prompt message. The first prompt message is used to prompt authorization to collect the voice data input by the target account in the current application, and the collected voice data is used to train the target speech synthesis model corresponding to the target account. In response to the confirmation operation of the first prompt information, a second prompt information is displayed, the second prompt information including preset text, the second prompt information being used to prompt the input of voice data according to the preset text; the input voice data is acquired; if the acquired voice data matches the preset text, the voice data acquisition permission of the target account is acquired, the voice data acquisition permission indicating that the voice data input based on the target account in the current application is allowed to be used to train the target speech synthesis model; If the voice data collection permission of the target account is obtained, the voice data input by the target account in the current application is obtained; The target speech synthesis model is trained according to the speech synthesis model training method according to any one of claims 1 to 7.

9. The method according to claim 8, characterized in that, The method further includes: The model display interface shows the status information of the target speech synthesis model, which indicates the training progress of the target speech synthesis model in multiple emotion types.

10. The method according to claim 9, characterized in that, The model display interface includes a text input area and an emotion selection area; the method also includes: In response to a speech synthesis instruction in the model display interface, the text information input in the text input area is obtained, and the emotion type selected in the emotion selection area is obtained; Based on the target speech synthesis model, the emotion type, and the text information, speech data that matches the text information and belongs to the emotion type is generated; Play the aforementioned audio data.

11. A speech synthesis model training device, characterized in that, The device includes: The acquisition module is used to acquire the sample spectrogram and sample label corresponding to the sample speech data, wherein the sample label indicates whether the sample speech data contains noise; The classification module is used to obtain the first speech feature corresponding to the sample spectrogram based on the feature extraction model; classify the first speech feature to obtain a predicted label, the predicted label indicating the possibility that the sample speech data contains noise; The determination module is used to determine a second gradient information that is inversely related to the first gradient information based on the first gradient information of the loss value of this training with respect to the first speech feature, wherein the loss value is determined based on the sample label and the predicted label; A generation module is used to determine, based on a phoneme clustering model, similar phoneme features of each sample phoneme feature within the first speech feature from multiple standard phoneme features contained in the phoneme mapping table; construct a second speech feature from the similar phoneme features of all sample phoneme features; process the second speech feature to obtain a second predicted spectrogram; and process the phoneme sequence corresponding to the sample speech data and the first speech feature based on a speech synthesis model to obtain a first predicted spectrogram, wherein the phoneme sequence includes at least one phoneme, and the pronunciation of the at least one phoneme matches the sound in the sample speech data. The training module is used to train the speech synthesis model, the feature extraction model, the phoneme clustering model, and the phoneme mapping table based on the difference between the sample spectrogram and the first predicted spectrogram, the difference between the sample spectrogram and the second predicted spectrogram, and the second gradient information.

12. A voice data acquisition device, characterized in that, The device includes: The display module is used to display the model training interface. The model training interface displays a first prompt message. The first prompt message is used to prompt authorization to collect the voice data input by the target account in the current application, and the collected voice data is used to train the target speech synthesis model corresponding to the target account. The acquisition module is configured to respond to a confirmation operation of the first prompt information by displaying a second prompt information, the second prompt information including preset text, the second prompt information being used to prompt the input of voice data according to the preset text; acquire the input voice data; and, if the acquired voice data matches the preset text, acquire the voice data acquisition permission of the target account, the voice data acquisition permission indicating that the voice data input based on the target account in the current application is allowed to be used to train the target speech synthesis model; The acquisition module is also used to acquire the voice data input by the target account in the current application when the target account has acquired the voice data acquisition permission; The target speech synthesis model is trained according to the speech synthesis model training method according to any one of claims 1 to 7.

13. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to perform the operations performed by the speech synthesis model training method as described in any one of claims 1 to 7; or to perform the operations performed by the speech data acquisition method as described in any one of claims 8 to 10.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to perform the operations performed by the speech synthesis model training method as described in any one of claims 1 to 7; or to perform the operations performed by the speech data acquisition method as described in any one of claims 8 to 10.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it performs the operations performed by the speech synthesis model training method as described in any one of claims 1 to 7; or, it performs the operations performed by the speech data acquisition method as described in any one of claims 8 to 10.

Images