Speech synthesis method and device, storage medium and electronic device

Abstract

The embodiment of the invention provides a speech synthesis method and device, a storage medium and an electronic device, and the method comprises the steps: inputting an input text and a specified target identity into an audio output model, extracting an identity feature sequence of the target identity through an identity recognition model, extracting a phoneme feature sequence corresponding to the input text through a coding layer of the speech synthesis model, superposing the identity feature sequence of the target identity and the phoneme feature sequence corresponding to the input text, and inputting the superposed identity feature sequence and the superposed phoneme feature sequence into a variable adapter of the speech synthesis model, and performing duration prediction, alignment, energy prediction and pitch prediction on the phoneme feature sequence through the variable adapter, outputting target Mel spectrum features corresponding to the input text through a decoding layer of a speech synthesis model, inputting the target Mel spectrum features into a vocoder, and outputting the target Mel spectrum features corresponding to the input text. And outputting a synthetic audio according with the target identity speaking style through the vocoder.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a speech synthesis method and device, a storage medium, and an electronic device. Background Art

[0002] Text To Speech (TTS) technology in related technologies is a technology that converts text into speech and outputs it. As the single-speaker TTS method becomes more and more mature, the multi-speaker TTS method is further promoted. Different from the single-speaker TTS method, the multi-speaker TTS method usually needs to use a speaker embedding to encode the voice, style, accent and other features of each speaker in order to distinguish the different characteristics of each speaker.

[0003] In many cases, the synthesized speech of the existing multi-speaker TTS model will have a neutral voice style (for example, if there are two speakers corresponding to a male speaker and a female speaker, ideally, the two speakers correspond to a male voice style and a female voice style respectively. Voice style neutralization means that the two styles influence each other, making the final speech synthesis effect tend to be neutral).

[0004] In the related art, the speech synthesis method based on multiple speakers cannot solve the speech neutralization phenomenon, resulting in unsatisfactory speech synthesis effect. No effective solution has been proposed in the related art. Summary of the invention

[0005] The embodiments of the present application provide a speech synthesis method and device, a storage medium, and an electronic device to at least solve the problem in the related art that the speech synthesis method based on multiple speakers cannot solve the speech neutralization phenomenon, resulting in unsatisfactory speech synthesis effect.

[0006] In one embodiment of the present application, a speech synthesis method is provided, comprising: inputting input text and a specified target identity identifier into an audio output model, wherein the target identity identifier uniquely corresponds to the target identity, the audio output model is a neural network model trained using sample data, and the audio output model includes an identity recognition model and a speech synthesis model; extracting an identity feature sequence of the target identity through the identity recognition model, wherein the identity recognition model saves an identity mapping table during the training process, and the identity mapping table includes a mapping between the identity identifier and the identity feature sequence, the identity recognition model is a model constructed based on a residual network, and during the training process, a convolution residual module is used as the main body to extract input audio features, and then a preset number of Gaussian components is determined based on the input audio features; according to the preset number of Gaussian components, data in the input audio feature set is preprocessed to generate the input Gaussian components of data in the audio feature set are input; the information of each Gaussian component includes: weight information, vector information and covariance matrix; the phoneme feature sequence corresponding to the input text is extracted through the encoding layer of the speech synthesis model, wherein the encoding layer of the speech synthesis model sets different encoding methods for different types of languages; the identity feature sequence of the target identity and the phoneme feature sequence corresponding to the input text are superimposed and input into the variable adapter of the speech synthesis model, and the phoneme feature sequence is predicted and aligned in duration, predicted in energy and predicted in pitch by the variable adapter, and then decoded by the decoding layer of the speech synthesis model and the target Mel-spectrogram feature corresponding to the input text is output, wherein the target Mel-spectrogram feature conforms to the speaking style of the target identity; the target Mel-spectrogram feature is input into a vocoder, and the vocoder outputs a synthesized audio conforming to the speaking style of the target identity.

[0007] In one embodiment, during the training process of the identity recognition model, after extracting input audio features using a convolutional residual module as the main body, the frame-level input is converted into speaker features at the speech level through a feature averaging module, and the temporarily aggregated speaker features are mapped into an identity feature sequence corresponding to the speaker's identity identifier through a linear transformation and normalization module. The identity recognition model is trained through a triple loss function to maximize the cosine similarity between the same speakers and minimize the cosine similarity between different people. In one embodiment, before inputting the input text and the specified target identity identifier into the audio output model, the method includes: using a first training sample to train an initial model of the identity recognition model to obtain a first identity recognition model; using a second training sample to train the initial model of the speech synthesis model to obtain a first speech synthesis model; using a third training sample to simultaneously train the first identity recognition model and the first speech synthesis model to obtain the audio output model; or, using the first training sample to train the initial model of the identity recognition model to obtain the first identity recognition model, and then using the third training sample to train the first identity recognition model to obtain the identity recognition model; using the second training sample to train the initial model of the speech synthesis model to obtain the first speech synthesis model, and then using the third training sample to train the first speech synthesis model to obtain the speech synthesis model; wherein the first training sample and the second training sample are general training data, and the third training sample contains audio of the target identity group.

[0008] In one embodiment, the method of using the first training sample to train the initial model of the identity recognition model to obtain the first identity recognition model includes: inputting the first training sample into the initial model of the identity recognition model, wherein the first training sample includes: a first sample audio containing audio of multiple speakers and an identity label corresponding to each speaker, and the initial model of the identity recognition model includes: at least two convolution residual modules, an average module, an affine module, a normalization module and a loss function module, wherein the convolution residual module is composed of a first convolution layer and a first residual layer, and the first residual layer includes at least two convolution sublayers and an activation function sublayer; outputting the frame-level audio features corresponding to the first sample audio data through the convolution residual module, generating the Gaussian components of the data in the input audio feature set, and inputting the Gaussian components into the average module after zero mean and variance normalization, wherein, The information extraction of the Gaussian component refers to describing the feature sequence of the speech data with statistics, wherein the extracted statistics belong to high-dimensional features, and then projecting them into a low-dimensional space to obtain an identity parameter vector; performing average processing on the received data through the averaging module so that audio of a specific length corresponds to audio features of a specific length, and then inputting the averaged data into the affine module; performing dimensionality reduction processing on the received data through the affine module, and then mapping the dimensionality reduced data into an identity feature sequence representing the identity of the speaker through the standardization module; verifying the difference between the identity feature sequence and the identity label through the loss function module, and updating the model parameters of the initial model of the identity recognition model. After multiple iterations, when the difference between the identity feature sequence and the identity label is less than a first preset threshold, the first identity recognition model is obtained.

[0009] In one embodiment, the use of the second training sample to train the initial model of the speech synthesis model to obtain the first speech synthesis model includes: inputting the second training sample into the initial model of the speech synthesis model, wherein the second training sample includes: a second sample text and a second sample audio recorded by a speaker with the second sample text as the content, the initial model of the speech synthesis model includes: a feature encoding module, a variable adaptation module and a feature decoding module, the variable adaptation module includes a duration prediction layer, a length adjustment layer, an energy prediction layer and a pitch prediction layer; the phoneme feature sequence corresponding to the second sample text and the audio feature sequence corresponding to the second sample audio data, wherein the The phoneme feature sequence corresponding to the second sample text is extracted by the feature encoding module, and the audio feature sequence corresponding to the second sample audio data is processed in advance by the data preprocessing step; the phoneme feature sequence corresponding to the second sample text is subjected to time prediction processing by the time prediction layer, and the time prediction result is compared with the audio feature sequence corresponding to the second sample audio data to obtain a first loss function; the phoneme feature sequence corresponding to the second sample text and the audio feature sequence corresponding to the second sample audio data are input into the time prediction layer as first data, and the time prediction layer calculates the low-dimensional mean and variance of the first input data to obtain the second data , sampling the second data from the latent variable space to obtain potential data, performing dimensionality-upgrading processing on the potential data to obtain third data, comparing the first data with the third data to obtain a second loss function, and comparing the probability distribution of the third data with the standard normal distribution to obtain a third loss function, wherein the first loss function, the second loss function, and the third loss function jointly adjust the parameters of the duration prediction layer according to specific weights; inputting the data output by the duration prediction layer into the length adjustment layer, and performing alignment processing according to the Gaussian distribution probability during the data lengthening process; inputting the audio feature sequences corresponding to the second sample audio data into the energy prediction layer Output and the pitch prediction layer, output energy feature sequence and pitch feature sequence; input the data of the length adjustment layer, the energy prediction layer output and the pitch prediction layer output into the feature decoding module respectively, and output Mel spectrum features through the feature decoding module; verify the difference between the Mel spectrum features output by the feature decoding module and the audio feature sequence corresponding to the second sample audio data through the loss function, and update the model parameters of the initial model of the speech synthesis model. After multiple iterations, when the difference between the Mel spectrum features output by the feature decoding module and the audio feature sequence corresponding to the second sample audio data is less than a second preset threshold, the first speech synthesis model is obtained.

[0010] In one embodiment, the method of using the third training sample to simultaneously train the first identity recognition model and the first speech synthesis model to obtain the audio output model includes: inputting the third training sample into an initial model of the audio output model, wherein the initial model of the audio output model includes the first identity recognition model and the first speech synthesis model, and the third training sample includes: a third sample audio and a third sample text corresponding to the third sample audio; extracting a phoneme feature sequence from the third sample text through the first speech synthesis model, and converting the sequence into a phoneme latent variable feature through a latent variable space; extracting an identity feature of each of the third sample audios through the first identity recognition model, And encode the identity identifier corresponding to each identity feature to obtain the identity identification feature; superimpose the phoneme latent variable feature, the identity feature and the identity identification feature to obtain a latent variable sequence, and use the latent variable sequence to train the variable adapter module of the first speech synthesis model; verify the difference between the Mel-spectrogram feature output by the first speech synthesis model and the audio feature corresponding to the third sample audio data through a loss function, and update the model parameters of the initial model of the audio output model. After multiple iterations, when the difference between the Mel-spectrogram feature output by the first speech synthesis model and the audio feature corresponding to the third sample audio data is less than a third preset threshold, the audio output model is obtained.

[0011] In another embodiment of the present application, a speech synthesis device is also provided, including: an input module, configured to input input text and a specified target identity identifier into an audio output model, wherein the target identity identifier uniquely corresponds to the target identity, and the audio output model is a neural network model trained using sample data, and the audio output model includes an identity recognition model and a speech synthesis model; a first extraction module, configured to extract an identity feature sequence of the target identity through the identity recognition model, wherein the identity recognition model saves an identity mapping table during the training process, and the identity mapping table includes a mapping between the identity identifier and the identity feature sequence, and the identity recognition model is a model constructed based on a residual network, and during the training process, a convolution residual module is used as the main body to extract input audio features, and then a preset number of Gaussian components is determined based on the input audio features; according to the preset number of Gaussian components, the data in the input audio feature set is preprocessed to generate the input Gaussian components of data in an audio feature set; information of each of the Gaussian components includes: weight information, vector information and covariance matrix; a second extraction module, configured to extract the phoneme feature sequence corresponding to the input text through the encoding layer of the speech synthesis model, wherein the encoding layer of the speech synthesis model sets different encoding methods for different types of languages; a processing module, configured to superimpose the identity feature sequence of the target identity and the phoneme feature sequence corresponding to the input text and input them into the variable adapter of the speech synthesis model, and after performing duration prediction and alignment, energy prediction and pitch prediction on the phoneme feature sequence through the variable adapter, input it into the decoding layer of the speech synthesis model and output a target Mel-spectrogram feature corresponding to the input text, wherein the target Mel-spectrogram feature conforms to the speaking style of the target identity; an output module, configured to input the target Mel-spectrogram feature into a vocoder, and output a synthesized audio conforming to the speaking style of the target identity through the vocoder.

[0012] In one embodiment of the present application, a computer-readable storage medium is further provided, wherein a computer program is stored in the storage medium, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0013] In one embodiment of the present application, an electronic device is also proposed, including a memory and a processor, characterized in that a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any of the above method embodiments.

[0014] Through the speech synthesis method provided in the embodiment of the present application, the input text and the specified target identity identifier are input into the audio output model, the identity feature sequence of the target identity is extracted through the identity recognition model, the phoneme feature sequence corresponding to the input text is extracted through the encoding layer of the speech synthesis model, the identity feature sequence of the target identity and the phoneme feature sequence corresponding to the input text are superimposed and input into the variable adapter of the speech synthesis model, the phoneme feature sequence is subjected to duration prediction and alignment, energy prediction and pitch prediction through the variable adapter, and then passes through the decoding layer of the speech synthesis model and outputs the target Mel-spectrogram feature corresponding to the input text, the target Mel-spectrogram feature is input into the vocoder, and the vocoder outputs the synthesized audio that conforms to the speaking style of the target identity. During the training process, the convolution residual module is used as the main body to extract the input audio features, and then the preset number of Gaussian components is determined according to the input audio features; according to the preset number of Gaussian components, the data in the input audio feature set is preprocessed to generate the Gaussian components of the data in the input audio feature set, which solves the problem that the speech synthesis method based on multiple speakers in the related art cannot solve the speech neutralization phenomenon, resulting in unsatisfactory speech synthesis effect. The identity mapping table is saved in the training process through the identity recognition model, and the identity mapping table contains the mapping between the identity identifier and the identity feature sequence. Then, during the audio synthesis process, the identity feature sequence of the target identity can be extracted through the identity recognition model, and the identity feature sequence of the target identity and the phoneme feature sequence corresponding to the input text are superimposed and input into the variable adapter of the speech synthesis model. Then, after the phoneme feature sequence is predicted and aligned in duration, predicted in energy and predicted in pitch, it passes through the decoding layer of the speech synthesis model and outputs the target Mel spectrum feature corresponding to the input text. At this time, the target Mel spectrum feature conforms to the speaking style of the target identity, effectively avoids the speech neutralization phenomenon, and the synthesized audio is closer to the speaking style of the target identity. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flowchart of an optional speech synthesis method according to an embodiment of the present application; Figure 2 is a structural diagram of an optional audio output model according to an embodiment of the present application; Figure 3 is a schematic structural diagram of an optional speech synthesis device according to an embodiment of the present application; Figure 4 It is a schematic diagram of an optional electronic device structure according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0017] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0018] Figure 1 is a flowchart of an optional speech synthesis method according to an embodiment of the present application, such as Figure 1 As shown, the embodiment of the present application provides a speech synthesis method, including: Step S102, inputting the input text and the specified target identity into an audio output model, wherein the target identity uniquely corresponds to the target identity, and the audio output model is a neural network model trained using sample data, and the audio output model includes an identity recognition model and a speech synthesis model; Step S104, extracting the identity feature sequence of the target identity through the identity recognition model, wherein the identity recognition model saves an identity mapping table during the training process, and the identity mapping table contains the mapping between the identity identifier and the identity feature sequence, and the identity recognition model is a model constructed based on a residual network. During the training process, the convolution residual module is used as the main body to extract the input audio features, and then the preset number of Gaussian components is determined according to the input audio features; according to the preset number of Gaussian components, the data in the input audio feature set is preprocessed to generate Gaussian components of the data in the input audio feature set; the information of each Gaussian component includes: weight information, vector information and covariance matrix; Step S106, extracting a phoneme feature sequence corresponding to the input text through a coding layer of a speech synthesis model, wherein the coding layer of the speech synthesis model sets different coding modes for different types of languages; Step S108, superimposing the identity feature sequence of the target identity and the phoneme feature sequence corresponding to the input text and inputting them into the variable adapter of the speech synthesis model, performing duration prediction and alignment, energy prediction and pitch prediction on the phoneme feature sequence through the variable adapter, decoding through the decoding layer of the speech synthesis model and outputting the target Mel-spectrogram feature corresponding to the input text, wherein the target Mel-spectrogram feature conforms to the speaking style of the target identity; Step S110: input the target mel-spectrogram feature into a vocoder, and output a synthesized audio that conforms to the target identity's speaking style through the vocoder.

[0019] It should be noted that the input text can be in various languages ​​such as Chinese, English, Korean, Japanese or French. The encoding layer of the speech synthesis model can set different encoding methods for different types of languages ​​to obtain accurate phoneme feature sequences.

[0020] It should be noted that during the training process, the identity recognition model and the speech synthesis model are first trained separately and then trained together. For example, a large number of training samples in a general database can be used to train the initial identity recognition model, and then the training samples containing specific identities can be used to train the identity recognition model. Similarly, a large number of training samples in a general database can be used to train the initial speech synthesis model, and then the training samples containing specific identities can be used to train the speech synthesis model. When using training samples containing specific identities to train the identity recognition model and the speech synthesis model, the two models can be trained together directly, or they can be trained separately and then trained together.

[0021] In one embodiment, during the training process of the identity recognition model, after extracting input audio features using a convolutional residual module as the main body, the frame-level input is converted into speaker features at the speech level through a feature averaging module, and the temporarily aggregated speaker features are mapped into an identity feature sequence corresponding to the speaker's identity identifier through a linear transformation and normalization module. The identity recognition model is trained through a triple loss function to maximize the cosine similarity between the same speakers and minimize the cosine similarity between different people.

[0022] It should be noted that the identity recognition model is a model built on a residual network, which maps the speech sentence to a hyperplane, and then calculates the similarity between speakers through cosine similarity, thereby distinguishing the characteristics of different speakers. After the speech passes through the identity recognition model, it is trained in conjunction with a speech synthesis model based on a conformer structure to improve the similarity between the speech of multiple speakers and the real speaker.

[0023] Figure 2 is a schematic diagram of the structure of an optional audio output model according to an embodiment of the present application, such as Figure 2As shown, in one embodiment, before inputting the input text and the specified target identity identifier into the audio output model, the method includes: using the first training sample to train the initial model of the identity recognition model to obtain the first identity recognition model; using the second training sample to train the initial model of the speech synthesis model to obtain the first speech synthesis model; using the third training sample to simultaneously train the first identity recognition model and the first speech synthesis model to obtain the audio output model; or, using the first training sample to train the initial model of the identity recognition model to obtain the first identity recognition model, and then using the third training sample to train the first identity recognition model to obtain the identity recognition model; using the second training sample to train the initial model of the speech synthesis model to obtain the first speech synthesis model, and then using the third training sample to train the first speech synthesis model to obtain the speech synthesis model, wherein the first training sample and the second training sample are general training data, and the third training sample contains audio of the target identity group.

[0024] It should be noted that the first training sample and the second training sample are obtained from a general database; the third training sample is data containing audio of the target identity group. The first training sample contains a large amount of multi-speaker speech data, so that the trained first identity recognition model has a generalized function, can effectively identify different speaker identity identifiers, match the corresponding identity feature sequence, and then match the corresponding speaking style.

[0025] In one embodiment, the method of using the first training sample to train the initial model of the identity recognition model to obtain the first identity recognition model includes: inputting the first training sample into the initial model of the identity recognition model, wherein the first training sample includes: a first sample audio containing audio of multiple speakers and an identity label corresponding to each speaker, and the initial model of the identity recognition model includes: at least two convolution residual modules, an average module, an affine module, a normalization module and a loss function module, wherein the convolution residual module is composed of a first convolution layer and a first residual layer, and the first residual layer includes at least four convolution sublayers and one activation function sublayer; outputting the frame-level audio features corresponding to the first sample audio data through the convolution residual module, generating the Gaussian components of the data in the input audio feature set, and inputting the Gaussian components into the average module after zero mean and variance normalization, wherein, The information extraction of the Gaussian component refers to describing the feature sequence of the speech data with statistics, wherein the extracted statistics belong to high-dimensional features, and then projecting them into a low-dimensional space to obtain an identity parameter vector; performing average processing on the received data through the averaging module so that audio of a specific length corresponds to audio features of a specific length, and then inputting the averaged data into the affine module; performing dimensionality reduction processing on the received data through the affine module, and then mapping the dimensionality reduced data into an identity feature sequence representing the identity of the speaker through the standardization module; verifying the difference between the identity feature sequence and the identity label through the loss function module, and updating the model parameters of the initial model of the identity recognition model. After multiple iterations, when the difference between the identity feature sequence and the identity label is less than a first preset threshold, the first identity recognition model is obtained.

[0026] It should be noted that the convolution residual module is mainly composed of the first convolution layer CNN and the first residual layer Res. The convolution residual module can include 2, 3, 4 or even more. The more the number, the better the training result. The embodiment of the present application preferably uses 3 or 5. The first residual layer Res can be stacked by 2 to 4 residual blocks, and each residual block is composed of 2 to 4 3*3 convolution sublayers (step size is 1) and 1 Relu activation function. The convolution residual module is mainly used to extract features at the speech frame level, which are features related to the speaker. For example, 64-dimensional mfcc features can be extracted, and zero mean and variance normalization can be performed.

[0027] It should be noted that the related processing involved in extracting the features at the speech frame level by the convolution residual module may include: determining the preset number of Gaussian components; preprocessing the data in the identity recognition data set according to the preset number of Gaussian components to generate Gaussian components of the data in the identity recognition data set; wherein the information of each Gaussian component includes: weight information, vector information and covariance matrix.

[0028] Specifically, in the initialization process of the training process of the first identity recognition model, M=2048 Gaussian component components can be set. That is, 2048 single Gaussian components are generated, and each i component (i=1, 2...M) includes a weight, a mean vector, and a covariance matrix: λ = {ci, μi, ∑i}, where μi and ∑i are vectors and matrices respectively.

[0029] Gaussian component information extraction refers to describing the feature sequence of speech data, such as MFCC feature sequence, with statistics. The extracted statistics belong to high-dimensional features, which are then projected into low-dimensional space to obtain identity parameter vectors. In practical applications, identity parameter vectors represent the identity information of different speakers.

[0030] The average module is used to implement the operation of averaging speech features. Its main function is to average the speech frame features from the convolution residual module so that one segment of speech corresponds to one segment of features, rather than one segment of speech corresponding to one segment of features for each frame of speech. It mainly performs dimensionality reduction and normalization to reduce computational complexity, because a segment of speech has many frames N, and each frame corresponds to a 66-dimensional feature, which will result in a small segment of speech having N*66 dimensions, which is very computationally complex and will be very slow to train. The affine module is used to transform feature dimensions, such as changing a feature of dimension 2048 to 512 dimensions, thereby reducing the dimension and further reducing computational complexity, reducing the number of model parameters, and avoiding the situation where training is difficult to converge. The normalization module normalizes the reduced-dimensional features and maps them into a vector representing the speaker's speech, that is, a sequence of identity features representing the speaker's identity.

[0031] It should be noted that the difference between the identity feature sequence and the identity tag is verified by the loss function module, and the model parameters of the initial model of the identity recognition model are updated. In the embodiment of the present application, the loss function for training the identity recognition model is a triple loss function. Three samples are simultaneously input into the identity recognition model, an anchor point (a sentence "M" said by a specific speaker A), a positive example (another sentence P said by the same speaker A), and a negative example (a sentence "N" said by another speaker B). What B said can be the same as what A said, or it can be different. The cosine similarity between the anchor point and the positive and negative examples is continuously updated to maximize the cosine similarity between the same speakers and minimize the cosine similarity between different people. After multiple iterations, when the difference between the identity feature sequence and the identity tag is less than the first preset threshold, the first identity recognition model is obtained. The calculation of cosine similarity is calculated according to the existing formula cosine_similarity(). The input is two values ​​to be compared (three samples are compared two by two), such as M and N here, and the output corresponds to A's identity tag.

[0032] In one embodiment, the use of the second training sample to train the initial model of the speech synthesis model to obtain the first speech synthesis model includes: inputting the second training sample into the initial model of the speech synthesis model, wherein the second training sample includes: a second sample text and a second sample audio recorded by a speaker with the second sample text as the content, the initial model of the speech synthesis model includes: a feature encoding module, a variable adaptation module and a feature decoding module, the variable adaptation module includes a duration prediction layer, a length adjustment layer, an energy prediction layer and a pitch prediction layer; the phoneme feature sequence corresponding to the second sample text and the audio feature sequence corresponding to the second sample audio data, wherein the The phoneme feature sequence corresponding to the second sample text is extracted by the feature encoding module, and the audio feature sequence corresponding to the second sample audio data is processed in advance by the data preprocessing step; the phoneme feature sequence corresponding to the second sample text is subjected to time prediction processing by the time prediction layer, and the time prediction result is compared with the audio feature sequence corresponding to the second sample audio data to obtain a first loss function; the phoneme feature sequence corresponding to the second sample text and the audio feature sequence corresponding to the second sample audio data are input into the time prediction layer as first data, and the time prediction layer calculates the low-dimensional mean and variance of the first input data to obtain the second data , sampling the second data from the latent variable space to obtain potential data, performing dimensionality-upgrading processing on the potential data to obtain third data, comparing the first data with the third data to obtain a second loss function, and comparing the probability distribution of the third data with the standard normal distribution to obtain a third loss function, wherein the first loss function, the second loss function, and the third loss function jointly adjust the parameters of the duration prediction layer according to specific weights; inputting the data output by the duration prediction layer into the length adjustment layer, and performing alignment processing according to the Gaussian distribution probability during the data lengthening process; inputting the audio feature sequences corresponding to the second sample audio data into the energy prediction layer Output and the pitch prediction layer, output energy feature sequence and pitch feature sequence; input the data of the length adjustment layer, the energy prediction layer output and the pitch prediction layer output into the feature decoding module respectively, and output Mel spectrum features through the feature decoding module; verify the difference between the Mel spectrum features output by the feature decoding module and the audio feature sequence corresponding to the second sample audio data through the loss function, and update the model parameters of the initial model of the speech synthesis model. After multiple iterations, when the difference between the Mel spectrum features output by the feature decoding module and the audio feature sequence corresponding to the second sample audio data is less than a second preset threshold, the first speech synthesis model is obtained.

[0033] It should be noted that the audio feature sequence corresponding to the second sample audio data is processed in advance by a data preprocessing step, and the data preprocessing step mainly includes: extracting the audio Mel spectrum, pitch, energy, and duration prediction features used in training.

[0034] It should be noted that the phoneme feature sequence corresponding to the second sample text is extracted by the feature encoding module. The feature encoding module encodes the phonemes in the training data into phoneme vectors and adds position encoding to express the correlation between the phonemes and the information of their positions.

[0035] It should be noted that the variable adapter can introduce additional acoustic feature information, such as fundamental frequency (pitch), duration, energy, etc. to control the quality of the synthesized speech. In speech, the duration of the phoneme directly affects the length of the pronunciation and the overall rhythm; the fundamental frequency is another feature that affects emotion and rhythm; the energy affects the amplitude of the spectrum and directly affects the volume of the audio, so the introduction of these three variables will improve the quality of the synthesized audio. Since the length of the Mel spectrum of the audio is much larger than the length of the input phoneme, the output needs to be equal in length, so the length adjustment layer is needed to use the duration information to expand the length of the encoder output to the spectrum length. The extended length can be achieved by directly upsampling according to the specific value of the duration.

[0036] It should be noted that the duration prediction unit is used to predict the duration of the sample text. The duration is used to control the rhythm and tempo of the synthesized speech from the text, such as the rhythm and speed of speaking, which can make the synthesized speech more natural. The duration prediction unit further includes an MFA (Montreal Forced Aligner) subunit and a variational autoencoding subunit, and the two subunits are independent of each other.

[0037] The MFA subunit is used to perform duration prediction processing on the phoneme sequence output by the feature encoding module, compare the duration prediction result with the sample audio duration, and obtain a first loss function.

[0038] The variational autoencoder subunit is used to autoencode the text and audio in the sample, obtain a new data by autoencoding the features, and obtain the second loss function by comparing the audio obtained by autoencoding with the sample audio. The core content of the variational autoencoder subunit is to use the potential features to calculate the loss function. The following describes the working principle of the variational autoencoder subunit: The variational auto-encoding subunit is composed of an encoder part and a decoder part. The encoder part is used to calculate the low-dimensional (about several hundred dimensions) mean and variance of the input data X (features of phoneme features and audio features spliced ​​together, equivalent to the first data), and further sample X from the latent variable space through the low-dimensional mean and variance (to obtain the second data) to obtain the potential data Z (equivalent to the potential data) corresponding to the input data. The decoder part is used to generate new data Y (equivalent to the third data) after dimensionality upgrade based on the above potential data Z. The principle of the variational auto-encoding subunit is to obtain the potential data Z by sampling according to the probability distribution of the input data X, and further generate new data Y that is different from the original input data X but follows the probability distribution of the input data X. In this way, in the process of speech synthesis, even if the number of samples is small, the probability distribution of the sample data can be obtained through the above self-encoding process to obtain a better training effect.

[0039] In the above latent variable space sampling process, in order to ensure that the probability distribution of the potential data Z conforms to the input data X, we can first assume that there is a posterior probability of Z with respect to X, and further assume that the posterior probability distribution conforms to the normal distribution. The posterior probability distribution is used to calculate the prior probability of Z with respect to X, so that the probability distribution of Z can be made to conform to the probability distribution of the input data X.

[0040] For the second loss function, it includes two parts, one part is used to calculate the similarity between the generated data Y and the original data X, and the other part is used to calculate the difference between the probability distribution of the output data Y and the standard normal distribution.

[0041] The first loss function of the MFA subunit and the second loss function of the variational autoencoder subunit together constitute the loss function of the duration prediction layer in the embodiment of the present application. The outputs of the MFA subunit and the variational autoencoder subunit are uniformly input into the length adjustment layer for processing.

[0042] In one embodiment, the method of using a third training sample to simultaneously train the first identity recognition model and the first speech synthesis model to obtain the audio output model includes: inputting the third training sample into an initial model of the audio output model, wherein the initial model of the audio output model includes the first identity recognition model and the first speech synthesis model, and the third training sample includes: a third sample audio and a third sample text corresponding to the third sample audio, wherein the third sample audio includes audio of a target identity group; extracting a phoneme feature sequence from the third sample text through the first speech synthesis model, and converting the sequence into phoneme latent variable features through a latent variable space; extracting each of the third sample texts through the first identity recognition model; The method comprises the following steps: obtaining an identity feature of the sample audio, encoding the identity identifier corresponding to each identity feature to obtain an identity identifier feature; superimposing the phoneme latent variable feature, the identity feature and the identity identifier feature to obtain a latent variable sequence, and using the latent variable sequence to train the variable adapter module of the first speech synthesis model; verifying the difference between the Mel-spectrogram feature output by the first speech synthesis model and the audio feature corresponding to the third sample audio data through a loss function, and updating the model parameters of the initial model of the audio output model. After multiple iterations, when the difference between the Mel-spectrogram feature output by the first speech synthesis model and the audio feature corresponding to the third sample audio data is less than a third preset threshold, the audio output model is obtained.

[0043] It should be noted that during joint training, the training text is first passed through the feature encoding module in the first speech synthesis model to obtain a phoneme feature sequence, and at the same time, each identity feature sequence of the training data is extracted using the trained first identity recognition model, and the identifiers of multiple speakers are encoded to obtain an identity identification feature sequence. At the same time, the fundamental frequency features, duration features, and energy features in the audio data are extracted, and these three are combined and superimposed with the phoneme feature sequence, identity feature sequence, and identity identification feature sequence, so that latent variables with the speaker's timbre are obtained, and then the latent variables are input into the variable adapter module for training together, and finally the adapted latent variables generated by the variable adapter module are converted into Mel-spectrogram features in parallel through the decoder.

[0044] In a specific application, the input text and the specified target identity are input into the audio output model, and after the decoder outputs the target mel-spectrogram features, the vocoder can output the synthesized audio that conforms to the speaking style of the target identity.

[0045] In another embodiment of the present application, a speech synthesis device is provided for implementing any of the above method steps, and the solutions already described are not repeated here. Figure 3 As shown, the speech synthesis device comprises: An input module 302 is configured to input an input text and a specified target identity into an audio output model, wherein the target identity uniquely corresponds to the target identity, the audio output model is a neural network model trained using sample data, and the audio output model includes an identity recognition model and a speech synthesis model; The first extraction module 304 is configured to extract the identity feature sequence of the target identity through the identity recognition model, wherein the identity recognition model saves an identity mapping table during the training process, and the identity mapping table contains a mapping between an identity identifier and an identity feature sequence. The identity recognition model is a model constructed based on a residual network. During the training process, a convolution residual module is used as the main body to extract input audio features, and then a preset number of Gaussian components is determined according to the input audio features; according to the preset number of Gaussian components, data in the input audio feature set are preprocessed to generate Gaussian components of the data in the input audio feature set; information of each Gaussian component includes: weight information, vector information and covariance matrix; A second extraction module 306 is configured to extract a phoneme feature sequence corresponding to the input text through the encoding layer of the speech synthesis model, wherein the encoding layer of the speech synthesis model sets different encoding methods for different types of languages; The processing module 308 is configured to superimpose the identity feature sequence of the target identity and the phoneme feature sequence corresponding to the input text and input them into the variable adapter of the speech synthesis model, and after performing duration prediction and alignment, energy prediction and pitch prediction on the phoneme feature sequence through the variable adapter, input them into the decoding layer of the speech synthesis model and output the target Mel-spectrogram feature corresponding to the input text, wherein the target Mel-spectrogram feature conforms to the speaking style of the target identity; The output module 310 is configured to input the target Mel-spectrogram feature into a vocoder, and output a synthesized audio that conforms to the speaking style of the target identity through the vocoder.

[0046] The audio output model provided in the embodiment of the present application can generate a general model after training with the first sample data and the second sample data. During use, the user only needs to provide a small sample data of 10-20 minutes to complete the model training containing the targeted target identity. The combination of the variational autoencoding subunit and the MFA subunit enables a relatively ideal training effect to be achieved through the above-mentioned small sample data of 10-20 minutes.

[0047] According to another aspect of the embodiment of the present application, an electronic device for implementing the above-mentioned speech synthesis method is also provided, and the above-mentioned electronic device can be but is not limited to being applied in a server. Figure 4As shown, the electronic device includes a memory 402 and a processor 404. The memory 402 stores a computer program, and the processor 404 is configured to execute the steps in any of the above method embodiments through the computer program.

[0048] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0049] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program: S1, inputting the input text and the specified target identity into an audio output model, wherein the target identity uniquely corresponds to the target identity, the audio output model is a neural network model trained using sample data, and the audio output model includes an identity recognition model and a speech synthesis model; S2, extracting the identity feature sequence of the target identity through the identity recognition model, wherein the identity recognition model saves an identity mapping table during the training process, the identity mapping table contains the mapping between the identity identifier and the identity feature sequence, the identity recognition model is a model constructed based on a residual network, during the training process, the convolution residual module is used as the main body to extract the input audio features, and then the preset number of Gaussian components is determined according to the input audio features; according to the preset number of Gaussian components, the data in the input audio feature set is preprocessed to generate the Gaussian components of the data in the input audio feature set; the information of each Gaussian component includes: weight information, vector information and covariance matrix; S3, extracting a phoneme feature sequence corresponding to the input text through the encoding layer of the speech synthesis model, wherein the encoding layer of the speech synthesis model sets different encoding methods for different types of languages; S4, superimposing the identity feature sequence of the target identity and the phoneme feature sequence corresponding to the input text and inputting them into the variable adapter of the speech synthesis model, performing duration prediction and alignment, energy prediction and pitch prediction on the phoneme feature sequence through the variable adapter, decoding the sequence through the decoding layer of the speech synthesis model and outputting the target Mel-spectrogram feature corresponding to the input text, wherein the target Mel-spectrogram feature conforms to the speaking style of the target identity; S5, inputting the target mel-spectrogram feature into a vocoder, and outputting a synthesized audio that conforms to the speaking style of the target identity through the vocoder.

[0050] Alternatively, a person skilled in the art may understand that: Figure 4The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 4 The structure of the electronic device is not limited. Figure 4 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 4 Different configurations are shown.

[0051] Among them, the memory 402 can be used to store software programs and modules, such as the program instructions / modules corresponding to the speech synthesis method and device in the embodiment of the present application, and the processor 404 executes various functional applications and data processing by running the software programs and modules stored in the memory 402, that is, realizing the above-mentioned speech synthesis method. The memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 402 may further include a memory remotely arranged relative to the processor 404, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof. Among them, the memory 402 can be specifically, but not limited to, used to store the program steps of the speech separation method.

[0052] Optionally, the transmission device 406 is used to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one example, the transmission device 406 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices and routers via a network cable so as to communicate with the Internet or a local area network. In one example, the transmission device 406 is a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0053] In addition, the electronic device further includes: a display 408 for displaying the speech synthesis process; and a connection bus 410 for connecting various module components in the electronic device.

[0054] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0055] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps: S1, inputting the input text and the specified target identity into an audio output model, wherein the target identity uniquely corresponds to the target identity, the audio output model is a neural network model trained using sample data, and the audio output model includes an identity recognition model and a speech synthesis model; S2, extracting the identity feature sequence of the target identity through the identity recognition model, wherein the identity recognition model saves an identity mapping table during the training process, the identity mapping table contains the mapping between the identity identifier and the identity feature sequence, the identity recognition model is a model constructed based on a residual network, during the training process, the convolution residual module is used as the main body to extract the input audio features, and then the preset number of Gaussian components is determined according to the input audio features; according to the preset number of Gaussian components, the data in the input audio feature set is preprocessed to generate the Gaussian components of the data in the input audio feature set; the information of each Gaussian component includes: weight information, vector information and covariance matrix; S3, extracting a phoneme feature sequence corresponding to the input text through the encoding layer of the speech synthesis model, wherein the encoding layer of the speech synthesis model sets different encoding methods for different types of languages; S4, superimposing the identity feature sequence of the target identity and the phoneme feature sequence corresponding to the input text and inputting them into the variable adapter of the speech synthesis model, performing duration prediction and alignment, energy prediction and pitch prediction on the phoneme feature sequence through the variable adapter, decoding the sequence through the decoding layer of the speech synthesis model and outputting the target Mel-spectrogram feature corresponding to the input text, wherein the target Mel-spectrogram feature conforms to the speaking style of the target identity; S5, inputting the target mel-spectrogram feature into a vocoder, and outputting a synthesized audio that conforms to the speaking style of the target identity through the vocoder.

[0056] The real-time audio-driven lip-sync system involved in the embodiments of the present application focuses on the back-end algorithm. During the implementation process, no other execution entities other than the server or terminal device used for computing and processing are involved. Only the commonly used video acquisition equipment, audio receiving equipment, video display equipment, etc. in the field are involved in the information collection and display stages.

[0057] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0058] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0059] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0060] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0061] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0062] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0063] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A speech synthesis method, characterized in that: include: Inputting the input text and the specified target identity into the audio output model, wherein the target identity uniquely corresponds to the target identity, the audio output model is a neural network model trained using sample data, and the audio output model includes an identity recognition model and a speech synthesis model; The identity feature sequence of the target identity is extracted through the identity recognition model, wherein the identity recognition model saves an identity mapping table during the training process, the identity mapping table contains a mapping between an identity identifier and an identity feature sequence, the identity recognition model is a model constructed based on a residual network, during the training process, a convolution residual module is used as the main body to extract input audio features, and then a preset number of Gaussian components is determined according to the input audio features; according to the preset number of Gaussian components, data in the input audio feature set are preprocessed to generate Gaussian components of the data in the input audio feature set; the information of each Gaussian component includes: weight information, vector information and covariance matrix; Extracting a phoneme feature sequence corresponding to the input text through the encoding layer of the speech synthesis model, wherein the encoding layer of the speech synthesis model sets different encoding methods for different types of languages; The identity feature sequence of the target identity and the phoneme feature sequence corresponding to the input text are superimposed and input into the variable adapter of the speech synthesis model, and the phoneme feature sequence is subjected to duration prediction and alignment, energy prediction and pitch prediction by the variable adapter, and then decoded by the decoding layer of the speech synthesis model and a target Mel-spectrogram feature corresponding to the input text is output, wherein the target Mel-spectrogram feature conforms to the speaking style of the target identity; The target mel-spectrogram features are input into a vocoder, and the vocoder outputs a synthesized audio that conforms to the speaking style of the target identity.

2. The speech synthesis method according to claim 1, characterized in that: During the training process of the identity recognition model, after the convolution residual module is used as the main body to extract the input audio features, the frame-level input is converted into the speaker features at the speech level through the feature averaging module, and the temporarily aggregated speaker features are mapped into an identity feature sequence corresponding to the speaker's identity identifier through the linear transformation and normalization module. The identity recognition model is trained through the triple loss function to maximize the cosine similarity between the same speakers and minimize the cosine similarity between different people.

3. The speech synthesis method according to claim 1, wherein: Before inputting the input text and the specified target identity into the audio output model, the method includes: Using the first training sample to train the initial model of the identity recognition model to obtain the first identity recognition model; using the second training sample to train the initial model of the speech synthesis model to obtain the first speech synthesis model; using the third training sample to simultaneously train the first identity recognition model and the first speech synthesis model to obtain the audio output model; or Using the first training sample to train the initial model of the identity recognition model to obtain the first identity recognition model, and then using the third training sample to train the first identity recognition model to obtain the identity recognition model; using the second training sample to train the initial model of the speech synthesis model to obtain the first speech synthesis model, and then using the third training sample to train the first speech synthesis model to obtain the speech synthesis model; The first training sample and the second training sample are general training data, and the third training sample contains audio of a target identity group.

4. The speech synthesis method according to claim 3, characterized in that: The step of using the first training sample to train the initial model of the identity recognition model to obtain the first identity recognition model includes: Inputting the first training sample into the initial model of the identity recognition model, wherein the first training sample includes: a first sample audio containing audios of multiple speakers and an identity tag corresponding to each speaker, and the initial model of the identity recognition model includes: at least two convolution residual modules, an average module, an affine module, a normalization module and a loss function module, the convolution residual module is composed of a first convolution layer and a first residual layer, and the first residual layer includes at least four convolution sublayers and one activation function sublayer; Outputting the frame-level audio features corresponding to the first sample audio data through the convolution residual module, generating Gaussian components of the data in the input audio feature set, and performing zero mean and variance normalization before inputting the Gaussian components into the averaging module, wherein the information extraction of the Gaussian components refers to describing the feature sequence of the speech data with statistics, the extracted statistics belonging to high-dimensional features, and then projecting them into a low-dimensional space to obtain an identity parameter vector; The received data is averaged by the average module so that the audio of a specific length corresponds to the audio feature of a specific length, and then the averaged data is input into the affine module; The affine module performs dimensionality reduction processing on the received data, and then maps the dimensionality-reduced data into an identity feature sequence representing the speaker's identity through the standardization module; The difference between the identity feature sequence and the identity tag is verified by the loss function module, and the model parameters of the initial model of the identity recognition model are updated. After multiple iterations, when the difference between the identity feature sequence and the identity tag is less than a first preset threshold, the first identity recognition model is obtained.

5. The speech synthesis method according to claim 3, characterized in that: The method of using the second training sample to train the initial model of the speech synthesis model to obtain a first speech synthesis model includes: Inputting the second training sample into the initial model of the speech synthesis model, wherein the second training sample includes: a second sample text and a second sample audio recorded by a speaker with the second sample text as the content, and the initial model of the speech synthesis model includes: a feature encoding module, a variable adaptation module and a feature decoding module, and the variable adaptation module includes a duration prediction layer, a length adjustment layer, an energy prediction layer and a pitch prediction layer; Obtaining a phoneme feature sequence corresponding to the second sample text and an audio feature sequence corresponding to the second sample audio data, wherein the phoneme feature sequence corresponding to the second sample text is extracted by the feature encoding module, and the audio feature sequence corresponding to the second sample audio data is processed in advance by a data preprocessing step; Performing a duration prediction process on the phoneme feature sequence corresponding to the second sample text through the duration prediction layer, and comparing the duration prediction result with the audio feature sequence corresponding to the second sample audio data to obtain a first loss function; Inputting the phoneme feature sequence corresponding to the second sample text and the audio feature sequence corresponding to the second sample audio data as first data into the duration prediction layer, the duration prediction layer calculating the low-dimensional mean and variance of the first input data to obtain second data, sampling the second data from the latent variable space to obtain potential data, performing dimensionality upgrading on the potential data to obtain third data, comparing the first data with the third data to obtain a second loss function, and comparing the probability distribution of the third data with a standard normal distribution to obtain a third loss function, wherein the first loss function, the second loss function, and the third loss function jointly adjust the parameters of the duration prediction layer according to specific weights; The data output by the duration prediction layer is input into the length adjustment layer, and alignment processing is performed according to the Gaussian distribution probability in the process of data lengthening; Inputting the audio feature sequence corresponding to the second sample audio data into the energy prediction layer output and the pitch prediction layer respectively, and outputting an energy feature sequence and a pitch feature sequence; The data outputted by the length adjustment layer, the energy prediction layer and the pitch prediction layer are superimposed and inputted into the feature decoding module, and the Mel spectrum features are outputted through the feature decoding module; The difference between the Mel-spectrogram feature output by the feature decoding module and the audio feature sequence corresponding to the second sample audio data is verified by a loss function, and the model parameters of the initial model of the speech synthesis model are updated. After multiple iterations, when the difference between the Mel-spectrogram feature output by the feature decoding module and the audio feature sequence corresponding to the second sample audio data is less than a second preset threshold, the first speech synthesis model is obtained.

6. The speech synthesis method according to claim 3, characterized in that: The step of using the third training sample to simultaneously train the first identity recognition model and the first speech synthesis model to obtain the audio output model includes: Inputting the third training sample into the initial model of the audio output model, wherein the initial model of the audio output model includes the first identity recognition model and the first speech synthesis model, and the third training sample includes: a third sample audio and a third sample text corresponding to the third sample audio; Extracting a phoneme feature sequence from the third sample text through the first speech synthesis model, and converting the sequence into phoneme latent variable features through a latent variable space; Extracting the identity feature of each of the third sample audios through the first identity recognition model, and encoding the identity identifier corresponding to each identity feature to obtain an identity identifier feature; superimposing the phoneme latent variable features, the identity features and the identity identification features, as well as the fundamental frequency features, duration features and energy features corresponding to the third sample audio to obtain a latent variable sequence, and using the latent variable sequence to train the variable adapter module of the first speech synthesis model; The difference between the Mel-spectrogram feature output by the first speech synthesis model and the audio feature corresponding to the third sample audio data is verified through a loss function, and the model parameters of the initial model of the audio output model are updated. After multiple iterations, when the difference between the Mel-spectrogram feature output by the first speech synthesis model and the audio feature corresponding to the third sample audio data is less than a third preset threshold, the audio output model is obtained.

7. A speech synthesis device, characterized in that: include: An input module configured to input an input text and a specified target identity into an audio output model, wherein the target identity uniquely corresponds to the target identity, the audio output model is a neural network model trained using sample data, and the audio output model includes an identity recognition model and a speech synthesis model; A first extraction module is configured to extract the identity feature sequence of the target identity through the identity recognition model, wherein the identity recognition model saves an identity mapping table during the training process, and the identity mapping table contains a mapping between an identity identifier and an identity feature sequence. The identity recognition model is a model constructed based on a residual network. During the training process, a convolution residual module is used as the main body to extract input audio features, and then a preset number of Gaussian components is determined according to the input audio features; according to the preset number of Gaussian components, data in the input audio feature set are preprocessed to generate Gaussian components of the data in the input audio feature set; information of each Gaussian component includes: weight information, vector information and covariance matrix; A second extraction module is configured to extract a phoneme feature sequence corresponding to the input text through an encoding layer of the speech synthesis model, wherein the encoding layer of the speech synthesis model sets different encoding methods for different types of languages; A processing module is configured to superimpose the identity feature sequence of the target identity and the phoneme feature sequence corresponding to the input text and input them into the variable adapter of the speech synthesis model, and after performing duration prediction and alignment, energy prediction and pitch prediction on the phoneme feature sequence through the variable adapter, input them into the decoding layer of the speech synthesis model and output a target Mel-spectrogram feature corresponding to the input text, wherein the target Mel-spectrogram feature conforms to the speaking style of the target identity; The output module is configured to input the target Mel-spectrogram feature into a vocoder, and output a synthesized audio that conforms to the speaking style of the target identity through the vocoder.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.

Images