Model training method and device, speech synthesis method, device and storage medium

Abstract

The embodiments of the present application provide a model training method and apparatus, a speech synthesis method, device, and storage medium, and belong to the field of artificial intelligence technology. The method includes: obtaining and inputting training text information and training timbre information into a preset original speech synthesis model for speech synthesis processing to obtain preliminary feature information and speech spectrum prediction information; inputting the preliminary feature information into a preset original classification model for speaker discrimination to obtain preliminary speaker discrimination information; performing loss calculation on the speech spectrum prediction information and preliminary speaker discrimination information to obtain a target loss function; adjusting the parameters of the original classification model according to the target loss function to obtain a target classification model; obtaining and adjusting the parameters of the original speech synthesis model according to the target speaker discrimination information output by the target classification model to obtain a target speech synthesis model. The model trained by the embodiments of the present application can enhance the timbre expression of the speaker.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method and apparatus, a speech synthesis method, equipment, and storage medium. Background Art

[0002] Speech synthesis is a generative task that converts text phoneme sequences into acoustic features. Traditionally, speech synthesis technology utilizes the timbre characteristics of each speaker to jointly train speech synthesis for multiple speakers, leveraging this information to improve the training of the speech synthesis model. However, each speaker in the training sample set may have a different number of training samples. Excessive training samples for a single speaker can affect the similarity of the synthesized timbre, thereby impacting the model's training effectiveness. Summary of the Invention

[0003] The main purpose of the embodiments of the present application is to propose a model training method and device, a speech synthesis method, equipment and storage medium. The trained model can enhance the timbre expression of the speaking object, thereby improving the model training effect.

[0004] To achieve the above objectives, a first aspect of an embodiment of the present application proposes a model training method, the method comprising:

[0005] Acquire an original data set; wherein the original data set includes: speech synthesis training data and speaker verification information, and the speech synthesis training data includes: training text information, training timbre information and speech spectrum verification information;

[0006] Inputting the training text information and the training timbre information into a preset original speech synthesis model for speech synthesis processing to obtain preliminary feature information and speech spectrum prediction information;

[0007] Inputting the preliminary feature information into a preset original classification model to perform speaker identification to obtain preliminary speaker identification information;

[0008] performing loss calculation on the speech spectrum prediction information, the speech spectrum verification information, the preliminary speaker identification information, and the speaker verification information to obtain a target loss function;

[0009] Adjusting the parameters of the original classification model according to the target loss function to obtain a target classification model;

[0010] Obtaining target speaking object discrimination information output by the target classification model;

[0011] The parameters of the original speech synthesis model are adjusted according to the target speaker discrimination information to obtain a target speech synthesis model.

[0012] In some embodiments, the original speech synthesis model includes: a feature extraction network and a spectrum decoding network; the inputting the training text information and the training timbre information into a preset original speech synthesis model for speech synthesis processing to obtain preliminary feature information and speech spectrum prediction information includes:

[0013] Inputting the training text information and the training timbre information into the feature extraction network to perform feature extraction to obtain the preliminary feature information;

[0014] The preliminary feature information is input into the spectrum decoding network to select and splice speech segments to obtain the speech spectrum prediction information.

[0015] In some embodiments, the feature extraction network includes: a timbre feature extraction layer, a string processing layer, and an encoder; inputting the training text information and the training timbre information into the feature extraction network for feature extraction to obtain the preliminary feature information includes:

[0016] Inputting the training text information into the string processing layer for string processing to obtain training phoneme information;

[0017] Encoding the training phoneme information by the encoder to obtain a phoneme feature vector;

[0018] Inputting the training timbre information into the timbre feature extraction layer for feature extraction to obtain a timbre feature vector;

[0019] Attention processing is performed on the phoneme feature vector and the timbre feature vector to obtain the preliminary feature information.

[0020] In some embodiments, the spectrum decoding network includes: a first linear layer and a decoder; inputting the preliminary feature information into the spectrum decoding network to select and splice speech segments to obtain the speech spectrum prediction information includes:

[0021] Inputting the preliminary feature information into the first linear layer to perform feature correlation calculation to obtain first feature correlation information;

[0022] Selecting speech segments from the first feature association information by the decoder to obtain speech spectrum segments;

[0023] The decoder splices the speech spectrum segments to obtain the speech spectrum prediction information.

[0024] In some embodiments, the original classification model includes: a gradient reversal layer, a second linear layer, and a classification layer; inputting the preliminary feature information into the preset original classification model to perform speech object discrimination to obtain preliminary speech object discrimination information includes:

[0025] Inputting the preliminary feature information into the gradient inversion layer to perform gradient calculation to obtain linearly updated feature information;

[0026] Performing feature correlation calculation on the linearly updated feature information through the second linear layer to obtain second feature correlation information;

[0027] The second feature association information is classified by the classification layer to obtain the preliminary speaking object discrimination information.

[0028] In some embodiments, adjusting parameters of the original speech synthesis model according to the target speaker discrimination information to obtain a target speech synthesis model includes:

[0029] Performing loss calculation on the target speaker discrimination information and the speaker verification information to obtain a discrimination loss function;

[0030] Updating the target loss function according to the discriminant loss function to obtain an updated loss function;

[0031] The parameters of the original speech synthesis model are adjusted according to the updated loss function to obtain the target speech synthesis model.

[0032] To achieve the above-mentioned purpose, a second aspect of the embodiments of the present application provides a speech synthesis method, which includes:

[0033] Get target text information;

[0034] The target text information is input into a target speech synthesis model for speech synthesis to obtain target speech synthesis data; the target speech synthesis model is trained by the model training method as described in the first aspect.

[0035] To achieve the above objectives, a third aspect of the embodiments of the present application provides a model training device, comprising:

[0036] A data acquisition module is used to acquire an original data set; wherein the original data set includes: speech synthesis training data and speaker verification information, and the speech synthesis training data includes: training text information, training timbre information and speech spectrum verification information;

[0037] An information input module, configured to input the training text information and the training timbre information into a preset original speech synthesis model for speech synthesis processing, thereby obtaining preliminary feature information and speech spectrum prediction information;

[0038] a discrimination module, configured to input the preliminary feature information into a preset original classification model to perform speech object discrimination, thereby obtaining preliminary speech object discrimination information;

[0039] a loss calculation module, configured to perform loss calculation on the speech spectrum prediction information, the speech spectrum verification information, the preliminary speaker identification information, and the speaker verification information to obtain a target loss function;

[0040] A model adjustment module is used to adjust the parameters of the original classification model according to the target loss function to obtain a target classification model;

[0041] An information acquisition module, configured to acquire target speaking object discrimination information output by the target classification model;

[0042] The parameter adjustment module is used to adjust the parameters of the original speech synthesis model according to the target speaker discrimination information to obtain a target speech synthesis model.

[0043] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the model training method described in the first aspect, or the speech synthesis method described in the second aspect.

[0044] To achieve the above-mentioned purpose, the fifth aspect of an embodiment of the present application proposes a storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, it implements the model training method as described in the first aspect, or the speech synthesis method as described in the second aspect.

[0045] The model training method and apparatus, speech synthesis method, device and storage medium proposed in this application construct an original speech synthesis model and an original classification model into a generative adversarial network, that is, a GAN network, to first train the original classification model to obtain a target classification model, then obtain the target speaking object discrimination information output by the target classification model, and adjust the parameters of the original speech synthesis model according to the target speaking object discrimination information to obtain a target speech synthesis model. The constructed target speech synthesis model utilizes the distinction between different speaking objects, and the output speech synthesis data is more consistent with the speaking object, thereby improving the timbre similarity of the speaking object. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1This is a flow chart of the model training method provided in the embodiment of the present application;

[0047] Figure 2 yes Figure 1 Flowchart of step S102 in FIG.

[0048] Figure 3 yes Figure 2 Flowchart of step S201 in FIG.

[0049] Figure 4 yes Figure 2 Flowchart of step S202 in FIG.

[0050] Figure 5 yes Figure 1 Flowchart of step S103 in FIG.

[0051] Figure 6 yes Figure 1 Flowchart of step S107 in FIG.

[0052] Figure 7 is a flowchart of the speech synthesis method provided in an embodiment of the present application;

[0053] Figure 8 Schematic diagram of the structure of the model training device provided in the embodiment of the present application;

[0054] Figure 9 is a structural diagram of a speech synthesis device provided in an embodiment of the present application;

[0055] Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0057] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and so on, in the specification, claims, and drawings, are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0059] First, let’s analyze some of the terms used in this application:

[0060] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0061] Text-to-Speech (TTS): Text-to-speech (TSS) is a technology that produces artificial speech through mechanical and electronic means. Also known as text-to-speech (TTS), TTS can convert any text into standard, fluent speech in real time, effectively giving a machine an artificial mouth. It involves multiple disciplines, including acoustics, linguistics, digital signal processing, and computer science. It is a cutting-edge technology in Chinese information processing, primarily addressing the problem of converting text into audible sound—in other words, enabling machines to speak like humans.

[0062] Generative Adversarial Nets (GAN): Generative Adversarial Nets is a deep generative model based on adversarial learning. The main structure of the GAN includes a generator G (Generator) and a discriminator D (Discriminator). The entire training process is an adversarial game between the two: given Pdata(x), the goal is to learn G and D so that Pg = Pdata, so that D cannot correctly distinguish between the two.

[0063] Spectrum Prediction Network (Tacotron2): The entire feature prediction network of the spectrogram prediction network is a sequence-to-sequence network with an attention mechanism. Tacotron2 is an end-to-end speech synthesis framework. The model can be viewed from bottom to top as consisting of two parts: the spectrogram prediction network, an encoder-attention-decoder network, which predicts the input character sequence into a sequence of mel-spectrogram frames. The vocoder, a modified version of WaveNet, generates a time-domain waveform from the predicted mel-spectrogram sequence.

[0064] Gradient reversal layer (GRL): Gradient reversal is used in domain adaptation problems. The input x of the model passes through the encoder G f The feature vector f is obtained and then sent to two networks: (1) labeling classifier G y and (2) domain classifier G d The label classifier outputs the data label y, while the domain classifier predicts the domain d of the feature vector. f and domain classifier G d The training goal of is adversarial, so the article adds a gradient reversal layer GRL between the two. What GRL does is multiply the error passed to this layer by a negative number (-λ), which will make the training goals of the networks before and after GRL opposite to achieve the adversarial effect.

[0065] X-vector: Based on DNN encoding, X-vector aggregates the output of each speech segment after it passes through the model, rather than simply averaging it like D-vector. An upgraded version of D-vector, X-vector considers the entire audio signal during training. It calculates the mean and variance of the features output by each small segment, concatenates them, and then feeds them into a DNN to determine the speaker's identity.

[0066] Long Short-Term Memory (LSTM): A LSTM is a type of recurrent neural network (RNN) designed to address the vanishing and exploding gradient problems during long sequence training. An LSTM is a type of neural network containing LSTM blocks or other similar neural networks. In literature and other materials, LSTM blocks may be described as intelligent network units because they can remember values for indefinite periods of time. A gate within the block determines whether an input is important enough to be remembered and whether it can be output.

[0067] Cross Entropy: Cross entropy is a key concept in Shannon's information theory, primarily used to measure the difference between two probability distributions. Language model performance is typically measured using cross entropy and complexity. Cross entropy refers to the difficulty of using the model to recognize text, or from a compression perspective, the average number of bits required to encode each word. Complexity refers to the average number of branches used to represent the text using the model, the reciprocal of which can be considered the average probability of each word. Smoothing involves assigning a probability value to unobserved N-gram combinations to ensure that word sequences always receive a probability value through the language model.

[0068] Based on this, the embodiments of the present application provide a model training method and device, a speech synthesis method, equipment and storage medium, which first trains the original speech synthesis model to obtain a preliminary speech synthesis model, then fixes the parameters of the preliminary speech synthesis model, and adjusts the parameters of the original classification model according to the target loss function, so as to form a generative adversarial network through the original speech synthesis model and the original classification model, and alternately trains to obtain a target speech synthesis model. Even when the resources of the target speaker are relatively few, a target speech synthesis model that can distinguish the timbre of the target speaker can be trained, thereby improving the timbre similarity between the synthesized speech and the target speaker.

[0069] The model training method and apparatus, speech synthesis method, equipment and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the model training method in the embodiments of the present application is described.

[0070] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0071] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0072] The model training method and speech synthesis method provided in the embodiments of the present application relate to the field of artificial intelligence technology. The model training method and speech synthesis method provided in the embodiments of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the model training method and speech synthesis method, etc., but is not limited to the above forms.

[0073] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0074] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user voice data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0075] Figure 1 This is an optional flowchart of the model training method provided in the embodiment of the present application. Figure 1The method may include but is not limited to steps S101 to S107.

[0076] Step S101, obtaining an original data set; wherein the original data set includes: speech synthesis training data and speaker verification information, and the speech synthesis training data includes: training text information, training timbre information and speech spectrum verification information;

[0077] Step S102: inputting the training text information and the training timbre information into a preset original speech synthesis model for speech synthesis processing to obtain preliminary feature information and speech spectrum prediction information;

[0078] Step S103: inputting the preliminary feature information into a preset original classification model to perform speaker identification to obtain preliminary speaker identification information;

[0079] Step S104, performing loss calculation on the speech spectrum prediction information, the speech spectrum verification information, the preliminary speaker identification information, and the speaker verification information to obtain a target loss function;

[0080] Step S105, adjusting the parameters of the original classification model according to the target loss function to obtain a target classification model;

[0081] Step S106, obtaining target speaking object discrimination information output by the target classification model;

[0082] Step S107 , adjusting parameters of the original speech synthesis model according to the target speaker discrimination information to obtain a target speech synthesis model.

[0083] In steps S101 to S107 of the embodiment of the present application, speech synthesis training data and speaker verification information are obtained, wherein the speech synthesis training data includes training text information, training timbre information, and speech spectrum verification information. The training text information and training timbre information are input into an original speech synthesis model for speech synthesis processing to obtain preliminary feature information and speech spectrum prediction information. The preliminary feature information is input into an original classification model for speaker discrimination to obtain preliminary speaker discrimination information. A loss calculation is then performed on the preliminary speaker discrimination information, the speaker verification information, the speech spectrum prediction information, and the speech spectrum verification information to obtain a target loss function. Parameters of the original classification model are adjusted according to the target loss function to obtain a target classification model. Target speaker discrimination information output by the target classification model is then obtained, and parameters of the original speech synthesis model are adjusted according to the target speaker discrimination information to obtain a target speech synthesis model. Therefore, by constructing the original speech synthesis model and the original classification model into a generative adversarial network, that is, a GAN network, the original classification model is first trained to obtain the target classification model, and then the target speaking object discrimination information output by the target classification model is obtained. The parameters of the original speech synthesis model are adjusted according to the target speaking object discrimination information to obtain the target speech synthesis model. The constructed target speech synthesis model uses the distinction between different speaking objects, and the output speech synthesis data is more consistent with the speaking object, thereby improving the timbre similarity between the speech synthesis data output by the target speech synthesis model and the target speaking object.

[0084] In step S101 of some embodiments, an original dataset may be obtained from a training database. Information in the original dataset may also be collected in real time through other methods, without limitation, to update the original dataset in real time. The original dataset includes speech synthesis training data and speaker verification information, with each piece of speech synthesis training data corresponding to a speaker verification information. This means that the target speaker of the speech synthesis training data has been determined, and the speaker verification information can be used to determine the speaker of the speech synthesis training data.

[0085] See also Figure 2 In some embodiments, the original speech synthesis model includes: a feature extraction network and a spectrum decoding network; step S102 may include but is not limited to steps S201 to S202:

[0086] Step S201: input the training text information and the training timbre information into a feature extraction network to extract features and obtain preliminary feature information;

[0087] Step S202: input the preliminary feature information into a spectrum decoding network to select and splice speech segments to obtain speech spectrum prediction information.

[0088] In steps S201 to S202 shown in the embodiment of the present application, the training text information and the training timbre information are input into the feature extraction network to perform feature extraction to obtain preliminary feature information, and the preliminary feature information is then input into the spectrum decoding network. The spectrum decoding network then selects and splices speech segments based on the preliminary feature information to obtain speech spectrum prediction information. Therefore, by inputting the training text information and the training timbre information into the feature extraction network and the spectrum decoding network, speech spectrum prediction information can be obtained, making speech spectrum prediction simple. Wherein, the speech spectrum prediction information is the waveform information of the Mel spectrum, and the waveform information of the Mel spectrum can be directly input into the audio player to play the audio, making the speech synthesis operation simple.

[0089] In step S201 of some embodiments, the feature extraction network and the spectrum decoding network constitute Tacotron2, which is a spectrum prediction network that simplifies the steps of generating speech acoustic features and can improve the quality of speech spectrum generation. By inputting training text information and training timbre information into the feature extraction network, the feature extraction network extracts features from the training text information to obtain a text feature sequence, and extracts features from the training timbre information to obtain a timbre feature sequence. The text feature sequence and timbre feature sequence are then mapped into vectors of fixed length to output preliminary feature information, and the preliminary feature information is represented in the form of a vector.

[0090] In step S202 of some embodiments, a spectral decoding network selects matching speech segments based on the feature information, and then concatenates the speech segments into speech spectrum prediction information. The speech spectrum prediction information output by the spectral decoding network is a speech spectrum sequence, specifically a mel-spectrogram speech spectrum sequence.

[0091] See also Figure 3 In some embodiments, the feature extraction network includes: a timbre feature extraction layer, a string processing layer, and an encoder. Step S201 may include but is not limited to steps S301 to S304:

[0092] Step S301: input the training text information into the string processing layer for string processing to obtain training phoneme information;

[0093] Step S302: encoding the training phoneme information through an encoder to obtain a phoneme feature vector;

[0094] Step S303: input the training timbre information into the timbre feature extraction layer for feature extraction to obtain a timbre feature vector;

[0095] Step S304: Perform attention processing on the phoneme feature vector and the timbre feature vector to obtain preliminary feature information.

[0096] In steps S301 to S304 shown in the embodiment of the present application, training phoneme information is obtained by inputting training text information into the string processing layer for string processing, and then the training phoneme information is input into the encoder for encoding processing to obtain a phoneme feature vector, and the training timbre information is input into the timbre feature extraction layer for feature extraction to obtain a timbre feature vector, and the timbre feature vector and the timbre feature vector are subjected to attention processing to increase the weight of the associated features to obtain preliminary feature information, so that the generation of language acoustic features is simple.

[0097] In step S301 of some embodiments, the training text information is input into the string processing layer. The string processing layer performs string processing on the training text information, specifically dividing the training text information into multiple strings and constructing training phoneme information based on the vowels and consonants in each string. The training phoneme information is obtained by string processing the training text information. The training phoneme information is the smallest unit of speech and includes training vowel information and training consonant information, making it easy to obtain the training phoneme information.

[0098] In step S302 of some embodiments, the string processing layer outputs the training phoneme information and inputs the training phoneme information into an encoder. The encoder performs feature extraction and vector encoding on the phoneme information to obtain a phoneme feature vector. Specifically, the encoder includes four layers of two-dimensional convolutional layers. The training phoneme information is input into the four layers of two-dimensional convolutional layers in the form of a sequence and is mapped and converted using a preset vector to obtain a phoneme feature vector.

[0099] For example, the input training phoneme information is X=

x1,x2,x3,x4……,x Tx

[0100] F e =ReLU(F4*ReLU(F3*ReLU(F2*ReLU(F1*E(X))))) (1)

[0101] Where F1, F2, F3, and F4 are four two-dimensional convolutional layers, ReLU is the nonlinear activation on each convolutional layer, and E represents the vector processing of the training phoneme information X, that is, the embedding of the training phoneme information X.

[0102] In step S303 of some embodiments, the timbre feature extraction layer is a pre-trained X-vector model, which performs feature extraction on the training timbre information through the X-vector model to obtain a timbre feature vector, and uses the timbre feature vector to distinguish different speakers, thereby generating speech synthesis data for different speakers.

[0103] In step S304 of some embodiments, the phoneme feature vector and the timbre feature vector are added to obtain an updated feature vector, and then the attention mechanism is used to enhance the key feature vectors in the updated feature vector to calculate the attention influence relationship of different dimensions of the updated feature vector, and then different weight values are assigned according to the attention influence relationship to obtain preliminary feature information.

[0104] See also Figure 4 In some embodiments, the spectrum decoding network includes: a first linear layer and a decoder; step S202 may include but is not limited to steps S401 to S403:

[0105] Step S401: inputting the preliminary feature information into the first linear layer to perform feature correlation calculation to obtain first feature correlation information;

[0106] Step S402: Selecting speech segments from the first feature association information through a decoder to obtain speech spectrum segments;

[0107] Step S403: splicing the speech spectrum segments through a decoder to obtain speech spectrum prediction information.

[0108] In steps S401 to S403, as shown in the embodiment of the present application, preliminary feature information is input into a first linear layer to perform feature correlation calculation to obtain first feature correlation information. The decoder then selects speech segments from the first feature correlation information to obtain speech spectrum segments. Finally, the decoder concatenates the speech spectrum segments to obtain speech spectrum prediction information, which is mel-spectrogram waveform information. Therefore, by performing correlation calculation, speech segment selection, and concatenation on the preliminary feature information by the first linear layer and the decoder to obtain speech spectrum prediction information, the generation of speech spectrum prediction information is simplified.

[0109] In step S401 of some embodiments, the first linear layer is a fully connected layer, and the preliminary feature information is input into the fully connected layer to perform correlation calculation of all dimensions of the features to obtain first feature correlation information, so as to determine the correlation between the features through the first feature correlation information.

[0110] In step S402 of some embodiments, the first feature association information is input into a decoder. The embodiments of the present application employ the same decoder architecture as Tacotron2, comprising two LSTM layers and a linear layer. The decoder first selects speech segments based on the first feature association information to obtain speech spectrum segments that meet the first feature association information.

[0111] In step S403 of some embodiments, after obtaining the speech spectrum segments, the decoder splices the speech spectrum segments to obtain speech spectrum prediction information, and the speech spectrum prediction information is a spectrogram frame.

[0112] It should be noted that after the decoder outputs the speech spectrum prediction information, the speech spectrum prediction information is input into the convolution layer for convolution processing to generate speech spectrum prediction information with better quality.

[0113] See also Figure 5 In some embodiments, the original classification model includes: a gradient reversal layer, a second linear layer, and a classification layer; step S103 may include but is not limited to steps S501 to S503:

[0114] Step S501: input the preliminary feature information into the gradient inversion layer to perform gradient calculation to obtain linearly updated feature information;

[0115] Step S502: performing feature correlation calculation on the linearly updated feature information through a second linear layer to obtain second feature correlation information;

[0116] Step S503: Classify the second feature association information through the classification layer to obtain preliminary speaker identification information.

[0117] In steps S501 to S503 of the embodiment of the present application, preliminary feature information is output by the feature extraction network and input into the spectrum decoding network. The preliminary feature information is also input into the original classification model. The original speech synthesis model and the original classification model form a generative adversarial network (GAN) to continuously optimize the original speech synthesis model and the original classification model. The constructed speech synthesis model is capable of outputting speech synthesis data that conforms to the target speaker. Specifically, the preliminary feature information is first input into a gradient reversal layer for gradient calculation to obtain linearly updated feature information. A second linear layer then performs feature correlation calculation on the linearly updated feature information to obtain second feature correlation information. Finally, a classification layer performs classification processing on the second feature correlation information to obtain preliminary speaker discrimination information. This simplifies speaker discrimination and allows the determination of whether the speech spectrum prediction information generated by the original speech synthesis model conforms to the target speaker based on the preliminary speaker discrimination information. This allows the original speech synthesis model to be optimized, thereby improving the effectiveness of speech synthesis that conforms to the target speaker.

[0118] In step S501 of some embodiments, preliminary feature information is input into a gradient reversal layer for gradient calculation. Specifically, the speaker is determined based on the preliminary feature information, and different gradient calculation methods are used for different speakers to obtain linearly updated feature information. The gradient reversal layer uses two gradient calculation methods: one that negates the original gradient, and the other that takes the inverse of the original gradient. Thus, the gradient reversal layer outputs linearly updated feature information that can distinguish the target speaker.

[0119] In step S502 of some embodiments, the gradient flip layer outputs the linearly updated feature information to the second linear layer, and the second linear layer performs feature correlation calculation on the linearly updated feature information to obtain second feature correlation information, so as to determine the correlation between each feature of the linearly updated feature information according to the second feature correlation information.

[0120] In step S503 of some embodiments, the second linear layer outputs the second feature association information to the classification layer, which performs classification processing on the second feature association information. The classification layer then determines, based on the second feature association information, whether the preliminary feature information generated by the feature extraction network belongs to the target speaker, thereby obtaining preliminary speaker discrimination information. The preliminary speaker discrimination information includes target speaker information and non-target speaker information. The preliminary speaker discrimination information is used to determine whether the speech spectrum prediction information output by the original speech synthesis model matches the target speaker. The parameters of the original speech synthesis model are adjusted based on the preliminary speaker discrimination information to obtain a target speech synthesis model with improved speech synthesis performance.

[0121] In step S104 of some embodiments, a loss calculation is performed on the speech spectrum prediction information, the speech spectrum verification information, the preliminary speaker identification information, and the speaker verification information to obtain a target loss function. The target loss function includes a synthesis loss function and a discrimination loss function. The synthesis loss function is calculated based on the speech spectrum prediction information and the speech spectrum verification information to determine the error between the waveform information of the mel spectrum output by the original speech synthesis model and the waveform information of the true spectrum. The discrimination loss function is calculated based on the preliminary speaker identification information and the speaker verification information to determine the accuracy of the original classification model in speaker identification.

[0122] It should be noted that during the training process of the original speech synthesis model and the original classification model, the target loss function is continuously updated, and the parameters of the original speech synthesis model are adjusted according to the synthesis loss function. Since the preliminary feature information output after the parameters of the original speech synthesis model change also changes accordingly, the discriminant loss function of the original classification model also changes. The parameters of the original classification model are adjusted according to the discriminant loss function, and alternating parameter adjustments are performed until the target loss function converges, so as to determine the model parameters corresponding to the original speech synthesis model according to the converged target loss function.

[0123] In step S105 of some embodiments, after calculating the target loss function, the parameters of the original classification model are adjusted according to the target loss function to obtain a target classification model. Specifically, after performing a preset number of model training on the original speech synthesis model, the parameters of the original speech synthesis model are fixed, and preliminary feature information output by the parameters of the original speech synthesis model is input into the original classification model. A loss calculation is performed by calculating preliminary speaker discrimination information and speaker verification information output by the original classification model to obtain a discrimination loss function. The model parameters of the original classification model are adjusted according to the discrimination loss function to construct a target classification model that can accurately discriminate the speaker.

[0124] It should be noted that the cross entropy loss calculation is performed on the preliminary speaking object discrimination information and the speaking object verification information to obtain the first loss function and the second loss function. The first loss function is the probability of belonging to the target speaking object, and the second loss function is the probability of belonging to the non-target speaking object. The discrimination loss function is obtained by performing loss calculation on the first loss function and the second loss function. Among them, the calculation formula for calculating the cross entropy loss on the preliminary speaking object discrimination information and the speaking object verification information is shown in formula (2):

[0125]

[0126] In the formula, M represents the number of categories, y c It is a one-hot vector, c is the serial number of the category, and the element has only two values 0 and 1. If the category is the same as the sample category, it takes 1, otherwise it takes p c It represents the probability that the preliminary speaker identification information belongs to the speaker verification information. Therefore, the discriminant loss function of the original classification model is calculated by formula (2).

[0127] In step S106 of some embodiments, the preliminary feature information is input into the target classification model for speech object discrimination to obtain target speech object discrimination information, so as to determine whether the synthesized speech spectrum prediction information of the original speech synthesis model conforms to the target speech object based on the target speech object discrimination information.

[0128] It should be noted that the target classification model is equivalent to the discriminator in the GAN network, and requires alternating parameter adjustments with the original speech synthesis model. The feature extraction network fixes the parameters and then trains the original classification model to construct the target classification model. The target speaking object discrimination information output by the target classification model is used to train the original speech synthesis model, so that the target speaking object is considered during the speech synthesis model training process, and the speech synthesis data output by the constructed target speech synthesis model is more consistent with the timbre of the target speaking object.

[0129] See also Figure 6 In some embodiments, step S107 includes but is not limited to steps S601 to S603:

[0130] Step S601, performing loss calculation on the target speaker discrimination information and the speaker verification information to obtain a discrimination loss function;

[0131] Step S602, updating the target loss function according to the discriminant loss function to obtain an updated loss function;

[0132] Step S603: Adjust the parameters of the original speech synthesis model according to the updated loss function to obtain a target speech synthesis model.

[0133] In steps S601 to S603 shown in the embodiment of the present application, a discrimination loss function is obtained by performing loss calculation on the target speaking object discrimination information and the speaking object verification information, and the target loss function is updated according to the discrimination loss function to obtain an updated loss function. Then, the parameters of the original speech synthesis model are adjusted according to the updated loss function to obtain a target speech synthesis model. During the training process of the original speech synthesis model, the parameters of the original speech synthesis model are adjusted according to the discrimination information of whether the target speaking object is present, so as to construct a target speech synthesis model that can output speech synthesis data that is more consistent with the target speaking object, so as to enhance the expression of the target speaking object even when some speaking objects have limited resources, and improve the timbre similarity between the speech synthesis data output by the target speech synthesis model and the target speaking object.

[0134] In step S601 of some embodiments, a discriminant loss function is obtained by performing loss calculation on the target speaking object discrimination information and the speaking object verification information, and the discriminant loss function is used to determine whether the speech spectrum prediction information output by the original speech synthesis model is consistent with the target speaking object, so as to facilitate parameter adjustment of the original speech synthesis model.

[0135] In step S602 of some embodiments, the target loss function is updated using a discriminant loss function. Since parameter adjustments between the original classification model and the original speech synthesis model are performed alternately, when the initial feature information output by the original speech synthesis model changes after parameter adjustment, the discriminant loss function also changes accordingly, and the target loss function also changes synchronously. Therefore, after the parameters of the original speech synthesis model are adjusted, the parameters of the original classification model are adjusted to adjust the target loss, and when the target loss function converges, the target classification model is determined. By updating the discriminant loss function after constructing the target classification model, the target loss function is updated to obtain an updated loss function.

[0136] In step S603 of some embodiments, the parameters of the original speech synthesis model are adjusted by updating the loss function. By adjusting the parameters of the original speech synthesis model according to the updated loss function, target speaking object discrimination is added during the training process of the original speech synthesis model, so that the speech synthesis data output by the target speech synthesis model can be more consistent with the target speaking object. The trained model can enhance the timbre expression of the speaking object, so that a target speech synthesis model with better speech synthesis effect can be constructed for the target speaking object with a small amount of training resources.

[0137] The embodiment of the present application obtains training text information, training timbre information, speech spectrum verification information, and speaker verification information, and inputs the training text information into a string processing layer for string processing to obtain training phoneme information. The training phoneme information is encoded by an encoder to obtain a phoneme feature vector, and the training timbre information is input into a timbre feature extraction layer for feature extraction to obtain a timbre feature vector. Then, attention processing is performed on the phoneme feature vector and the timbre feature vector to obtain preliminary feature information, and the preliminary feature information is input into a first linear layer and a gradient flip layer. The preliminary feature information is subjected to feature correlation calculation by the first linear layer to obtain a first feature correlation, and then the first feature correlation is subjected to speech segment selection by the decoder to obtain speech spectrum segments, and the speech spectrum segments are spliced to obtain speech spectrum prediction information. At the same time, the gradient flip layer performs gradient calculation on the preliminary feature information to obtain linear update feature information, and then the second linear layer performs feature correlation calculation on the linear update feature information to obtain second feature correlation information, and then the classification layer performs classification processing on the second feature correlation information to obtain preliminary speaker discrimination information. A target loss function is obtained by performing loss calculation on preliminary speaking object discrimination information, speaking object verification information, speech spectrum prediction information and speech spectrum verification information, and the target classification model is obtained by adjusting the parameters of the original classification model according to the target loss function, so as to construct a target classification model that can accurately discriminate the speaking object, and then the target speaking object discrimination information output by the target classification model is obtained, and then the loss is calculated based on the target speaking object discrimination information and the speaking object verification information to obtain a discrimination loss function, and the target loss function is updated according to the discrimination loss function to obtain an updated loss function, and then the parameters of the original speech synthesis model are adjusted according to the updated loss function to obtain a target speech synthesis model, so as to construct a target speech synthesis model that can enhance the timbre expression of the target speaking object, and with a small amount of training resources for the target speaking object, the timbre similarity between the target speech synthesis data output by the target speech synthesis model and the target speaking object can be improved.

[0138] See also Figure 7 The present application also provides a speech synthesis method, which may include but is not limited to steps S701 to S702:

[0139] Step S701, obtaining target text information;

[0140] Step S702: Input the target text information into the target speech synthesis model for speech synthesis to obtain target speech synthesis data; the target speech synthesis model is trained by the model training method as described above.

[0141] In steps S701 to S702 shown in the embodiment of the present application, target text information is obtained and input into a target speech synthesis model so that the target text information is speech synthesized by the target speech synthesis model to obtain target speech synthesis data, thereby simplifying the speech synthesis operation. Furthermore, since the target speech synthesis model is obtained through the above-mentioned model training method, the output target speech synthesis data is more consistent with the timbre of the target speaker, thereby improving the timbre similarity of the target speaker.

[0142] In step S701 of some embodiments, the target text information may be obtained by recognizing an image, performing content recognition on audio data, or directly receiving the target text information input by the user. No specific limitation is imposed on the method of obtaining the target text information.

[0143] In step S702 of some embodiments, target text information is input into a target speech synthesis model. The target speech synthesis model then performs feature extraction on the target text information to obtain target phoneme feature information, encodes the target phoneme feature information to obtain a target phoneme feature vector, and obtains target timbre information. Feature extraction is then performed on the target timbre information to obtain a target timbre feature vector. The target timbre feature vector and the target timbre feature vector are then added together and attention processing is performed to obtain target feature information. The target feature information is input into a first linear layer for correlation analysis to obtain target feature association information. A decoder then uses the target feature association information to obtain target speech segments. The target speech segments are then concatenated to obtain target speech synthesis data. The target speech synthesis data is mel-spectrogram waveform data, and the audio corresponding to the target speech synthesis data can be directly played. Therefore, by inputting the target text information into the target speech synthesis model, target speech synthesis data that matches the target speaker is output, thereby improving the timbre similarity between the target synthesis data and the target speaker.

[0144] See also Figure 8 The present application also provides a model training device that can implement the above-mentioned model training method. The device includes:

[0145] The data acquisition module 801 is used to acquire an original data set; wherein the original data set includes speech synthesis training data and speaker verification information, and the speech synthesis training data includes training text information, training timbre information, and speech spectrum verification information;

[0146] The information input module 802 is used to input the training text information and the training timbre information into a preset original speech synthesis model for speech synthesis processing to obtain preliminary feature information and speech spectrum prediction information;

[0147] The discrimination module 803 is used to input the preliminary feature information into a preset original classification model to perform speech object discrimination and obtain preliminary speech object discrimination information;

[0148] a loss calculation module 804 for performing loss calculation on the speech spectrum prediction information, the speech spectrum verification information, the preliminary speaker identification information, and the speaker verification information to obtain a target loss function;

[0149] The model adjustment module 805 is used to adjust the parameters of the original classification model according to the target loss function to obtain the target classification model;

[0150] Information acquisition module 806, used to obtain target speaking object discrimination information output by the target classification model;

[0151] The parameter adjustment module 807 is used to adjust the parameters of the original speech synthesis model according to the target speaker discrimination information to obtain a target speech synthesis model.

[0152] The specific implementation of the model training device is basically the same as the specific embodiment of the above-mentioned model training method, and will not be repeated here.

[0153] See also Figure 9 The present invention also provides a speech synthesis device that can implement the above-mentioned speech synthesis method. The device includes:

[0154] The text acquisition module 901 is used to acquire target text information;

[0155] The speech synthesis module 902 is used to input the target text information into the target speech synthesis model for speech synthesis to obtain target speech synthesis data; the target speech synthesis model is trained by the model training device as described above.

[0156] The specific implementation of the speech synthesis device is basically the same as the specific embodiment of the above-mentioned speech synthesis method, and will not be repeated here.

[0157] The present application also provides an electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for enabling communication between the processor and the memory. When the program is executed by the processor, the aforementioned model training method or speech synthesis method is implemented. The electronic device may be any intelligent terminal, such as a tablet computer or an in-vehicle computer.

[0158] See also Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0159] The processor 1001 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0160] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program codes are stored in the memory 1002 and are called by the processor 1001 to execute the model training method or speech synthesis method of the embodiments of this application.

[0161] Input / output interface 1003, used to implement information input and output;

[0162] Communication interface 1004, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0163] Bus 1005 , which transmits information between various components of the device (e.g., processor 1001 , memory 1002 , input / output interface 1003 , and communication interface 1004 );

[0164] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0165] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to implement the above-mentioned model training method or speech synthesis method.

[0166] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0167] The model training method and apparatus, speech synthesis method, device, and storage medium provided in the embodiments of the present application are configured to input training text information and training timbre information into an original speech synthesis model for speech synthesis to obtain preliminary feature information and speech spectrum prediction information, then input the preliminary feature information into an original classification model for speech synthesis to obtain preliminary speaker discrimination information, and then perform a loss calculation on the speech spectrum prediction information, speech spectrum verification information, preliminary speaker discrimination information, and speaker verification information to obtain a target loss function. The parameters of the original classification model are then adjusted based on the target loss function to obtain a target classification model capable of accurately discriminating the speaker, and target speaker discrimination information of the target classification model is obtained. The parameters of the original speech synthesis model are then adjusted based on the target speaker discrimination information to incorporate target speaker discrimination into the training process of the original speech synthesis model, thereby constructing a target speech synthesis model trained to enhance the timbre expression of the speaker. The target speech synthesis data output by the constructed target speech synthesis model also better conforms to the target speaker.

[0168] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0169] It will be understood by those skilled in the art that Figure 1-7 The technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or a combination of certain steps, or different steps.

[0170] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0171] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0172] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0173] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0175] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0176] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0177] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0178] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A model training method, characterized in that: The method comprises: Acquire an original data set; wherein the original data set includes: speech synthesis training data and speaker verification information, and the speech synthesis training data includes: training text information, training timbre information and speech spectrum verification information; Inputting the training text information and the training timbre information into a preset original speech synthesis model for speech synthesis processing to obtain preliminary feature information and speech spectrum prediction information; Inputting the preliminary feature information into a preset original classification model to perform speaker identification to obtain preliminary speaker identification information; performing loss calculation on the speech spectrum prediction information, the speech spectrum verification information, the preliminary speaker identification information, and the speaker verification information to obtain a target loss function; Adjusting the parameters of the original classification model according to the target loss function to obtain a target classification model; Obtaining target speaking object discrimination information output by the target classification model; Adjusting parameters of the original speech synthesis model according to the target speaker discrimination information to obtain a target speech synthesis model; The step of adjusting parameters of the original speech synthesis model according to the target speaker discrimination information to obtain a target speech synthesis model includes: Performing loss calculation on the target speaker discrimination information and the speaker verification information to obtain a discrimination loss function; Updating the target loss function according to the discriminant loss function to obtain an updated loss function; The parameters of the original speech synthesis model are adjusted according to the updated loss function to obtain the target speech synthesis model.

2. The method according to claim 1, characterized in that The original speech synthesis model includes: a feature extraction network and a spectrum decoding network; the training text information and the training timbre information are input into the preset original speech synthesis model for speech synthesis processing to obtain preliminary feature information and speech spectrum prediction information, including: Inputting the training text information and the training timbre information into the feature extraction network to perform feature extraction to obtain the preliminary feature information; The preliminary feature information is input into the spectrum decoding network to select and splice speech segments to obtain the speech spectrum prediction information.

3. The method according to claim 2, characterized in that The feature extraction network includes: a timbre feature extraction layer, a string processing layer, and an encoder; the training text information and the training timbre information are input into the feature extraction network for feature extraction to obtain the preliminary feature information, including: Inputting the training text information into the string processing layer for string processing to obtain training phoneme information; Encoding the training phoneme information by the encoder to obtain a phoneme feature vector; Inputting the training timbre information into the timbre feature extraction layer for feature extraction to obtain a timbre feature vector; Attention processing is performed on the phoneme feature vector and the timbre feature vector to obtain the preliminary feature information.

4. The method according to claim 2, characterized in that The spectrum decoding network includes: a first linear layer and a decoder; inputting the preliminary feature information into the spectrum decoding network to select and splice speech segments to obtain the speech spectrum prediction information, including: Inputting the preliminary feature information into the first linear layer to perform feature correlation calculation to obtain first feature correlation information; Selecting speech segments from the first feature association information by the decoder to obtain speech spectrum segments; The decoder splices the speech spectrum segments to obtain the speech spectrum prediction information.

5. The method according to any one of claims 1 to 3, characterized in that The original classification model includes: a gradient reversal layer, a second linear layer and a classification layer; the inputting the preliminary feature information into the preset original classification model to perform speech object discrimination to obtain preliminary speech object discrimination information includes: Inputting the preliminary feature information into the gradient inversion layer to perform gradient calculation to obtain linearly updated feature information; Performing feature correlation calculation on the linearly updated feature information through the second linear layer to obtain second feature correlation information; The second feature association information is classified by the classification layer to obtain the preliminary speaking object discrimination information.

6. A speech synthesis method, characterized in that: The method comprises: Get target text information; The target text information is input into a target speech synthesis model for speech synthesis to obtain target speech synthesis data; the target speech synthesis model is trained by the model training method according to any one of claims 1 to 5.

7. A model training device, characterized in that: The model training method according to any one of claims 1 to 5, wherein the device comprises: A data acquisition module is used to acquire an original data set; wherein the original data set includes: speech synthesis training data and speaker verification information, and the speech synthesis training data includes: training text information, training timbre information and speech spectrum verification information; An information input module, configured to input the training text information and the training timbre information into a preset original speech synthesis model for speech synthesis processing, thereby obtaining preliminary feature information and speech spectrum prediction information; a discrimination module, configured to input the preliminary feature information into a preset original classification model to perform speech object discrimination, thereby obtaining preliminary speech object discrimination information; a loss calculation module, configured to perform loss calculation on the speech spectrum prediction information, the speech spectrum verification information, the preliminary speaker identification information, and the speaker verification information to obtain a target loss function; A model adjustment module is used to adjust the parameters of the original classification model according to the target loss function to obtain a target classification model; An information acquisition module, configured to acquire target speaking object discrimination information output by the target classification model; The parameter adjustment module is used to adjust the parameters of the original speech synthesis model according to the target speaker discrimination information to obtain a target speech synthesis model.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the model training method according to any one of claims 1 to 5, or the speech synthesis method according to claim 6.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the model training method according to any one of claims 1 to 5 or the speech synthesis method according to claim 6 is implemented.

Images