Information synthesis method, device, electronic device, and computer-readable storage medium

Abstract

Information synthesis method, information synthesis device, electronic device and computer-readable storage medium. According to the trained auto-codec network, a first target distribution is obtained, and according to the first target distribution, a target sampling code is obtained, and input information is encoded through a trained second encoder, a target code of the input information is obtained, and according to the target code, the target sampling code is decoded through a trained first decoder to obtain output information. Wherein, the first target distribution represents the characteristics of the posterior distribution from the input side to the output side of the first encoder, and during the training process of the auto-codec network, the input samples of the first encoder and the second encoder have one-to-one correspondence, and the prior distribution determined by the second encoder is used to predict the first target distribution. The above technical solution can be used in the field of speech synthesis, and the finally synthesized speech has better performance in prosody, pauses and character pronunciation.

Description

cross reference

[0001] This application claims priority to a Chinese patent application with application number 202011624655.8, titled "Information synthesis method, apparatus, electronic device and computer-readable storage medium," filed with the China Patent Office on December 30, 2020, the entire contents of which are hereby incorporated by reference into this application. [Technical field]

[0002] The present invention relates to the field of machine learning, and more particularly to an information synthesis method, an information synthesis device, an electronic device, and a computer-readable storage medium. [Background technology]

[0003] In recent years, with the development of information and the rise of artificial intelligence, human-computer interaction has become increasingly important, among which speech synthesis has become a hot topic in human-computer interaction research both at home and abroad.

[0004] Traditional speech synthesis includes three parts: front-end text analysis, parameter prediction, and vocoder. Among them, front-end text analysis mainly uses text information to predict speech features such as pronunciation sequence and prosodic boundaries, and usually requires the accumulation of pronunciation rules with sufficient text samples, as well as the prediction of pronunciation duration. After front-end text analysis, parameter prediction is mainly performed to predict various acoustic parameters, and the obtained prediction results are then input into the vocoder to perform voice synthesis and obtain speech information.

[0005] With the development of globalization and the deepening of exchanges between countries, multilingual synthesis has already become an increasingly important part of information communication and human-computer interaction. Traditional speech synthesis uses a processing method with a front end. That is, front-end text analysis requires professional linguistic engineers to process text information. However, in multilingual synthesis scenarios, it is necessary to grasp the phoneme dictionaries and pronunciation rules of different languages, and when these phoneme dictionaries and pronunciation rules are lacking, it is difficult to model the acoustic front end through front-end text analysis. In such scenarios, end-to-end TTS (text to speech) multilingual synthesis is recommended to use a "front-end" processing method that uses machine learning instead of linguistic engineers to process text information.

[0006] The front-end processing method requires the provision of a large amount of data to train the speech synthesis system, since there is no manual pre-analysis. In the conventional speech synthesis field, the data must have sufficiently high sound quality to prevent the speech synthesis model from being unable to accurately learn the speech features and pronunciation rules due to insufficient sample quality. However, it is difficult to obtain multilingual recording data that meets the sound quality requirements. The recording sample data used for synthesis may be less than one hour per language. In such cases, it is necessary to use data with poor recording quality. Since the recording quality data has various sources, the signal-to-noise ratio (SNR) of the data is low, and there are potential pronunciation errors, the end-to-end speech synthesis model is prone to errors in the training process, which leads to problems such as pronunciation overlap, mispronunciation, pronunciation ambiguity, and sound loss in the model.

[0007] Therefore, it is an urgent technical challenge for those skilled in the art to realize an efficient and accurate front-end speech synthesis solution using recording data with poor quantity and quality. This is the issue. Summary of the Invention

[0008] In view of this, the present application provides an information synthesis method, an information synthesis device, an electronic device, and a computer-readable medium that utilize recording data with poor quantity and quality to realize an efficient and accurate front-end voice synthesis solution.

[0009] The information synthesis method provided in the first aspect of the embodiment of the present invention includes: obtaining a first target distribution based on a trained automatic codec network, which represents the characteristics of the posterior distribution from the input side of the first encoder to the output side of the first encoder; obtaining a target sampling code based on the first target distribution; the trained second encoder encoding input information and obtaining a target code for the input information; and the trained first decoder decoding the target sampling code based on the target code and obtaining output information, wherein, during the training process of the automatic codec network, the input samples of the first encoder and the second encoder have a one-to-one correspondence, and the prior distribution determined by the second encoder is used to predict the first target distribution.

[0010] In an embodiment of the present application, the information synthesis method further includes training the first encoder, the second encoder, and the first decoder to obtain the trained first encoder, the trained second encoder, and the trained first decoder.

[0011] In an embodiment of the present application, before training the first encoder, the second encoder, and the first decoder, the information synthesis method further includes initializing the second encoder based on a pre-trained sample group that matches the type and input information of each sample.

[0012] In an embodiment of the present application, training the first encoder, the second encoder, and the first decoder includes: inputting a first sample group including a first sample and a second sample group including a second sample into the automatic codec network; and adjusting parameters of the first encoder, the second encoder, and the first decoder until the reconstruction loss of the automatic codec network for the first sample group meets a preset condition, wherein the type of the first sample matches the type of the output information, the type of the second sample matches the type of the input information, and the second sample has a one-to-one correspondence with the first sample.

[0013] In an embodiment of the present application, the reconstruction loss is obtained by the first encoder determining, based on each first sample, a first distribution parameter expressing a first distribution characteristic of a first code, which is a code of the first sample; obtaining a sampling code for each first sample based on the first distribution; encoding the second sample group by the second encoder to obtain a second code for the second sample corresponding to the first code of the first sample; obtaining, based on the second code, a second distribution parameter expressing a second distribution characteristic of the second code; obtaining a first dissimilarity between the first distribution and the second distribution based on a distribution parameter and the second distribution parameter; decoding a sampling code of the first sample based on a second code of a second sample corresponding to each first sample by the first decoder to obtain reconstructed samples of the first samples constituting a reconstructed sample group; obtaining a second dissimilarity between the first sample group and the reconstructed sample group based on the first sample group and the reconstructed sample group; and obtaining the reconstruction loss based on the first dissimilarity and the second dissimilarity. Wherein, the trained first encoder parameters are the first encoder parameters when the reconstruction loss satisfies a preset condition, and the first target distribution is the first distribution when the reconstruction loss satisfies the preset condition.

[0014] In an embodiment of the present application, the preset condition is that the reconstruction loss is a minimum value, or that the reconstruction loss is smaller than a preset value.

[0015] In an embodiment of the present application, in the process of adjusting parameters of the first encoder, the second encoder, and the first decoder until the reconstruction loss of the first sample group meets a preset condition, a first proportion of the first dissimilarities are backpropagated to the first encoder without backpropagating to the second encoder during gradient backpropagation, a second proportion of the first dissimilarities are backpropagated to the second encoder without backpropagating to the first encoder during gradient backpropagation, and the first proportion is smaller than the second proportion.

[0016] In an embodiment of the present application, the first dissimilarity measure is the relative entropy of the first distribution and the second distribution.

[0017] In an embodiment of the present application, the first distribution and the second distribution are the same type of probability distribution.

[0018] In an embodiment of the present application, the first encoder is the encoding side of a conditional variational autoencoder, and the first decoder is the decoding side of a conditional variational autoencoder.

[0019] In an embodiment of the present application, the second encoder is a transducer-based bidirectional encoder representation model.

[0020] In an embodiment of the present application, the input information is text and the output information is audio.

[0021] In an embodiment of the present application, the first decoder determines the audio information of the next frame based on the decoded historical output information in the audio and the context text in the text.

[0022] In an embodiment of the present application, when training the first encoder, the second encoder and the first decoder, it is not necessary to input phonemes and manually extracted text features to the first encoder and the second encoder.

[0023] The information synthesis device provided in the second aspect of the embodiment of the present invention is used in an auto-codec network including a first encoder, a second encoder, and a first decoder. The information synthesis device includes: a distribution acquisition module for obtaining a first target distribution representing the characteristics of the posterior distribution from the input side of the first encoder to the output side of the first encoder based on a trained auto-codec network; a sampling module for obtaining a target sampling code based on the first target distribution; an encoding module for encoding input information through the trained second encoder to obtain a target code of the input information; and a decoding module for decoding the target sampling code based on the target code through the trained first decoder to obtain output information, wherein, during the training process of the auto-codec network, the input samples of the first encoder and the second encoder are in one-to-one correspondence, and the prior distribution determined by the second encoder is used to predict the first target distribution.

[0024] In an embodiment of the present application, the information synthesis device further includes a training module for training the first encoder, the second encoder, and the first decoder to obtain the trained first encoder, the trained second encoder, and the trained first decoder.

[0025] In an embodiment of the present application, the information synthesis device further includes an initialization module for initializing the second encoder based on a pre-trained sample group in which each sample type and input information match.

[0026] An electronic device provided in a third aspect of an embodiment of the present invention includes a memory and a processor, the memory storing instructions, and when the instructions are executed by the processor, the device performs any one of the information synthesis methods described above.

[0027] A fourth aspect of an embodiment of the present invention provides a non-transitory computer readable storage medium storing instructions, which, when executed by the processor, perform any one of the information synthesis methods described above.

[0028] The speech synthesis method and speech synthesis device provided in the embodiments of the present application obtain a first target distribution based on a trained automatic codec network, obtain a target sampling code based on the first target distribution, a trained second encoder encodes input information, obtains a target code for the input information, and a trained first decoder decodes the target sampling code based on the target code to obtain output information. Wherein, the first target distribution represents the characteristics of the posterior distribution from the input side to the output side of the first encoder, and in the training process of the automatic codec network, the input samples of the first encoder and the second encoder correspond one-to-one, and the prior distribution determined by the second encoder is used to predict the first target distribution. Thus, the correspondence between the decoded output information and the input information highly restores (or simulates) the correspondence between the input samples of the first encoder and the second encoder in the training sample, so that the output information that corresponds well to the input information can be obtained. The present solution alleviates limitations on the quality of speech samples in the field of speech synthesis, especially in the field of multilingual speech synthesis, by obtaining a large number of training samples, so that the final synthesized speech has better performance in terms of prosody, pauses and character pronunciation, etc. [Brief description of the drawings]

[0029] In order to more clearly describe the technical solutions of the embodiments of the present invention or the prior art, the drawings necessary for describing the embodiments or the prior art will be briefly described below. Obviously, the drawings described below are only a part of the embodiments described in the present invention, and those skilled in the art can obtain other drawings according to the drawings provided without creative efforts.

[0030] [Figure 1] FIG. 1 is a flow chart showing a method for synthesizing information in an embodiment of the present application.

[0031] [Diagram 2] FIG. 2 is a flow chart illustrating another information synthesis method in an embodiment of the present application.

[0032] [Diagram 3] FIG. 3 is a flowchart showing another information synthesis method according to an embodiment of the present application.

[0033] [Figure 4] FIG. 4 is a diagram illustrating the flow of information when training the first decoder, the second decoder, and the first encoder in an embodiment of the present application.

[0034] [Diagram 5] FIG. 5 is a diagram showing the flow of information when performing information synthesis using the first decoder, the second decoder, and the first encoder trained in an embodiment of the present application.

[0035] [Figure 6] FIG. 6 is a diagram showing the information flow in a voice synthesis application scenario of the information synthesis method in the embodiment of the present application.

[0036] [Figure 7] FIG. 7 is a flowchart showing another information synthesis method according to an embodiment of the present application.

[0037] [Figure 8]FIG. 8 is a diagram showing the structure of another information synthesis device in an embodiment of the present application.

[0038] [Figure 9] FIG. 9 is a diagram showing the structure of another information synthesis device in an embodiment of the present application.

[0039] [Figure 10] FIG. 10 is a diagram showing the structure of another information synthesis device in an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0040] The technical solutions in the embodiments of the present invention will be described below in detail and in detail with reference to the drawings of the embodiments of the present invention. It is clear that the described embodiments are only a part of the embodiments of the present invention, and are not all of them. Based on the embodiments of the present invention, those skilled in the art can obtain all other embodiments without creative work, which are within the scope of protection of the present invention.

[0041] As used herein, relational terms such as first and second are used merely to distinguish one entity or operation from another entity or operation and do not necessarily require or imply that such an actual relationship or sequence exists between these entities or operations. Furthermore, the terms "comprise", "include", or other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or facility that includes a set of elements includes not only those elements, but also other elements not expressly listed or inherent to such process, method, article, or facility. In the absence of further limitations, an element qualified by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, article, or facility that includes the element.

[0042] Some of the terms used in the present invention will be explained below.

[0043] Encoder: The encoding side of an Autoencoder structure that maps input to code (latent variables, latent representations).

[0044] Decoder: The decoding side in an autoencoder structure that maps codes to outputs, usually a reconstruction of the input, possibly from the encoding side.

[0045] Tacotron model: An end-to-end general speech synthesis model based on an attention mechanism that can generate speech directly from text.

[0046] BERT model: Bidirectional Encoder Representations from Transformers is a deep, bidirectional, unsupervised sentence representation model that is pre-trained using only a plain text corpus. It is a pre-learning technology for natural language processing, announced by Google Inc. in 2018, and its purpose is to understand the meaning of users' search queries and extract prosodic information contained in the text.

[0047] VAE model: Variational Autoencoder is a generative model based on an autoencoder structure, which strengthens the assumptions on the latent variable distribution and uses the variational method. We learn the latent representation using , thereby introducing an additional loss component and a Stochastic Gradient Variational Bayes (SGVB) estimator into the training function.

[0048] One aspect of the present application provides a method for synthesizing information. In this embodiment, the information synthesis method is used in an automatic codec network including a first encoder, a second encoder and a first decoder model. The automatic codec network is realized by hardware, software or a combination of both. For example, the automatic codec network is placed in a hardware environment consisting of a server and a terminal, and a network connection is made between the server and the terminal, and the network includes but is not limited to a local area network (LAN), a metropolitan area network (MAN) or a wide area network (WAN). In this case, the above-mentioned encoder and decoder are implemented by software running on the server and the terminal, or by instructions embedded in the hardware on the server and the terminal. It should be noted that the above-mentioned encoder and decoder can run on the server or the terminal as a whole, and are not particularly limited in this embodiment.

[0049] The above information synthesis method will be further described below with reference to the steps shown in Figure 1. Figure 1 is a flow chart showing the information synthesis method in the embodiment of the present application. As shown in Figure 1, the information synthesis method includes the following steps:

[0050] S101: obtain a first target distribution based on the trained auto-codec network, where the first target distribution represents the feature of the posterior distribution from the input side of the first encoder to the output side of the first encoder, during the training process of the auto-codec network, the input samples of the first encoder and the second encoder have one-to-one correspondence, and the prior distribution determined by the second encoder is used to predict the first target distribution.

[0051] In this step, a first target distribution is first obtained from a trained autocodec network. Typically, the first target distribution is obtained from a first encoder part of the autocodec network and from a second encoder part. The first encoder and the second encoder may be artificial neural network models. For example, they may include one or more convolutional layers and / or pooling layers to extract features of information contained in the input. In this case, the first target distribution is regarded as a distribution of features extracted by the encoder, and its specific form is determined by the model of the first encoder and / or the second encoder.

[0052] The first target distribution may be a pre-defined type of distribution. For example, if the preset distribution is a normal distribution (also called Gaussian distribution), the obtained parameters can include the expectation value and standard deviation of the normal distribution, or the expectation value and standard deviation of the normal distribution can be obtained after calculating the obtained parameters. Of course, the preset type of distribution can be other types of distributions, as long as the first encoder and / or the second encoder determine their representations. Usually, the first target distribution is a multidimensional distribution.

[0053] It is understood that the training process of the auto-codec network includes the training process of the first encoder and the second encoder, that is, the parameters of the first encoder and the second encoder are iteratively adjusted. In the training process, the input samples of the first encoder and the input samples of the second encoder correspond one-to-one to ensure that the output information generated by the auto-codec network in the subsequent reconstruction process corresponds to the input information of the second encoder. The first encoder encodes its input samples, and the input samples are The second encoder can extract features of the input samples, and the extracted features follow a posterior distribution that depends on the first encoder model for the conditional probability of the input samples, i.e., the posterior distribution from the first encoder input side to the output side. The conditional probability here represents the probability of a particular feature (or a particular type of feature) obtained after inputting a particular input sample (or a particular type of input sample) to the first encoder, i.e., reflects the law between the first encoder input side and the output side. The second encoder can encode the input samples and extract features of the input samples, and these features follow a prior distribution determined by the input samples and the parameters of the second encoder. In training, the second encoder predicts the first target distribution, i.e., the parameters of the second encoder and the first encoder are constantly adjusted so that the prior distribution is mutually approximate to the posterior distribution from the first encoder input side to the output side. For example, the two become as similar as possible. Here, "as similar as possible" means that the difference between the two is below a certain threshold, or the difference between the two is a minimum value within a certain interval, or the difference between the two is a minimum value within a certain number of iteration steps during the training process. Please note that the specific embodiments of the present application are not limited thereto and may be determined according to actual usage needs.

[0054] After the training process is completed, a first target distribution is determined based on the trained first encoder and / or second encoder, and thus the first target distribution determined in the training process is obtained from these parameters in this step.

[0055] In one possible embodiment, the first encoder model is a VAE model. The first encoder model may be another encoder model including a neural network layer, and is not limited to a specific form here.

[0056] S102: Obtain a target sampling code based on the first target distribution.

[0057] After obtaining the first target distribution in S101, the first target distribution can be used to perform random sampling operation to obtain a sampling code. Of course, the parameters (such as the mean value) of the first target distribution can be directly used to obtain the target sampling code, and specifically, the parameters are sampled and encoded to obtain the target sampling code. It is understood that when the first target distribution is a multi-dimensional distribution, the obtained sampling code is a vector.

[0058] S103: The trained second encoder encodes the input information to obtain a target code for the input information.

[0059] Input information is input to a second encoder so that the second encoder encodes the input information. The second encoder may be an artificial neural network model. For example, it may include one or more convolutional layers and / or pooling layers, in which case the process of encoding the input information is equivalent to extracting necessary features therefrom. For example, if the input information is speech, speech features such as vocal prosody, pauses, and fundamental frequency may be extracted therefrom. If the input information is text, text features such as sentence division, word division, and meaning may be extracted therefrom. If the input information is an image, image features such as shape, color, and specific objects in the image may be extracted therefrom. The obtained target code may reflect information such as the strength of the extracted features.

[0060] In one possible embodiment, the second encoder model is a BERT model, in which case the input information may be text.

[0061] In this embodiment, the priority order for obtaining the target sampling code or the target code, i.e., S It should be noted that the execution order of S101, S102, and S103 is not limited. In one embodiment, the target sampling code can be obtained first, and then the target code, that is, S101 and S102 are executed, and then S103 is executed. In another embodiment, the target code can be obtained first, and then the target sampling code can be obtained, that is, S103 is executed, and then S101 and S102 are executed. In another embodiment, the target sampling code and the target code can be obtained simultaneously, that is, S103 is executed synchronously during the execution of S101 and S102.

[0062] For example, obtain the target code of the input information, input the target code to the fully connected layer, obtain the parameters of the first target distribution, fit the parameters of the first target distribution, obtain the first target distribution, sample the first target distribution, and obtain the target sampling code; or directly use the target code to obtain the first target distribution, and then sample the first target distribution to obtain the target sampling code.

[0063] S104: The trained first decoder decodes the target sampling code based on the target code to obtain output information.

[0064] After obtaining the target code and the target sampling code, they are input to the first decoder to complete the decoding process. The object of decoding is the target sampling code obtained by sampling the first target distribution, which represents the characteristics of the posterior distribution from the input side to the output side of the first encoder, so the output information is actually a new sample reconstructed by the first decoder with reference to the input sample during the training process of the first encoder, and the new sample and the input sample during the training of the first encoder have the same type. In addition, the decoding based on the target code is regarded as being conditioned on the target code. Since the target code is obtained by encoding the input information by the second encoder, the output information is actually constrained by the input information, i.e., corresponds to the input information. In the training process, the input of the first encoder and the input of the second encoder correspond one-to-one, and the prior distribution obtained after encoding by the second encoder is used to predict the first target distribution, so that the correspondence between the output information and the input information after decoding in this case can highly restore (or simulate) the correspondence between the input samples of the first encoder and the second encoder in the training sample. That is, the obtained output information can well correspond to the input information.

[0065] In the embodiment of the present application, the above information synthesis method is applied to speech synthesis technology. The input information is text, and the output information is speech. Therefore, the first encoder and the second encoder are the encoding side of the vocoder, and the first decoder is the decoding side of the vocoder. In the training process, one-to-one corresponding speech samples and text samples can be input into the first encoder and the second encoder, respectively, to extract speech features and text features respectively. The second encoder uses the prior distribution of the text features to predict the posterior distribution from speech to speech features, so that the first target distribution obtained from the trained automatic codec network is close to the distribution of the text features.

[0066] In conventional end-to-end speech synthesis solutions (such as the Tacotron model), speech synthesis based on its attention mechanism is subject to the interference of noise in the speech samples, which increases the requirements for recording quality and limits the number of training samples obtained, ultimately affecting the training results. In the technical proposal of the embodiment of the present application, since there is no speech noise in the text samples, the prediction of the first target distribution based on the prior distribution of text features in training can effectively mask the interference of noise contained in the speech samples on the finally generated speech, and obtain high-quality synthetic speech. In this way, the constraints on the quality of the speech samples during training are relaxed, a large number of training samples can be obtained, and the trained model has good performance in the prosody, pauses, and character pronunciation of the generated speech. Multilingual This advantage becomes even more pronounced in multilingual speech synthesis technology, since speech synthesis is highly dependent on the sample quality of each language.

[0067] In one embodiment, the first encoder and the first decoder are the encoding side model and the decoding side model of the VAE model, respectively, and the encoding result of the second encoder constrains the decoding process of the first decoder as a tag, i.e., the first encoder, the second encoder, and the first decoder together constitute a CVAE (Conditional Variational Autoencoder) structure.

[0068] In one embodiment, the second encoder is a BERT model. The front-end processing of conventional front-end sound synthesis solutions (such as the Tacotron model) is simple, and often uses convolution layers and long short-term memory networks. When the structural frame is simple and phonemes are used as front-end inputs, or when text features are extracted manually, good results can be obtained. However, when there is no front-end, only a few hundred hours of speech and corresponding text are available as training samples, making it difficult to understand the meaning of the text or capture the prosodic information of the sentence. In contrast, by pre-training a large amount of text, the BERT model scores very highly in the analysis of multilingual text, and has outstanding performance, especially in semantic understanding and semantic expression, thus overcoming the shortcomings of conventional front-end sound synthesis solutions.

[0069] From the above, the information synthesis method provided in the embodiment of the present application is based on a trained automatic codec network to obtain a first target distribution, and samples according to the first target distribution to obtain a target sampling code, and the trained second encoder encodes the input information to obtain a target code for the input information, and the trained first decoder uses the target code as a decoding condition to decode the target sampling code to obtain output information. Wherein, in the training process, the input samples of the first encoder and the second encoder correspond one-to-one, and the second encoder is used to predict the first target distribution. Thus, the correspondence between the decoded output information and the input information highly restores (or simulates) the correspondence between the input samples of the first encoder and the second encoder in the training sample, so that the output information that corresponds well to the input information can be obtained. This solution is to obtain a large number of training samples to alleviate the limitations on the quality of the voice samples in the field of speech synthesis, especially in the field of multilingual speech synthesis, so that the finally synthesized voice has better performance in prosody, pause and character pronunciation, etc.

[0070] 2 is a flow chart showing another information synthesis method in an embodiment of the present application. Based on the flow shown in FIG. 1, the information synthesis method further includes the following steps:

[0071] S105: Train a first encoder, a second encoder and a first decoder to obtain a trained first encoder, a trained second encoder and a trained first decoder.

[0072] From the description of S101, it can be seen that before the automatic codec network is applied to information synthesis, it is trained. Usually, the training is aimed at learning the encoding and decoding mapping network function and finding the preferred or most preferred codec network. In the training, usually, the training sample is inputted into the automatic codec network, and then, based on the training goal, the parameters of the first encoder, the second encoder and the first decoder are iteratively adjusted until the automatic codec network meets the preset condition on the training goal.

[0073] In one embodiment, the training goal can be set to reduce the reconstruction loss of samples in the automatic codec network. Therefore, the preset condition is specifically that the reconstruction loss of samples in the automatic codec network meets the preset condition. As shown in FIG. 3, in this case, S105 further includes the following steps:

[0074] S1051: Input a first sample group and a second sample group into an automatic codec network, where the first sample group includes a first sample, the type of the first sample matches the type of output information, the second sample group includes a second sample, the type of the second sample matches the type of input information, and the second sample has a one-to-one correspondence with the first sample.

[0075] It is understood that each part of the auto-codec network (such as the first encoder, the second encoder, and the first decoder) performs the same function in the information synthesis process and the training process, and the input and output sides of each correspond to the same type of information in both processes. Therefore, the type of the second sample input in the training process of the second encoder matches the type of the input information input in the information synthesis process. Although the input side of the first encoder is not directly used in the information synthesis process, during information synthesis, the decoding side (first decoder) of the auto-codec network decodes the target sampling code and obtains the output information. This is essentially to imitate the input side of the first encoder and reconstruct new samples. That is, the new samples (output information) must be similar to the samples input to the first encoder, that is, they must be of the same type, such as voice, text, image, etc. Therefore, in the training process, the auto-codec network reconstructs the first sample group, so the first sample input to the first encoder must also belong to this type.

[0076] S1052: Determine whether the reconstruction loss of the first sample group in the auto-codec network meets a preset condition. If the preset condition is met, execute S101 to S104 according to the above-mentioned embodiment. If the preset condition is not met, execute S1053.

[0077] S1053: Adjust the parameters of the first encoder, the second encoder, and the first decoder. After executing S1053, the process returns to S1052 for further determination.

[0078] In one embodiment, the preset condition is set so that the reconstruction loss is a minimum value. The minimum value here refers to the minimum reconstruction loss value within all parameter value ranges of the first encoder, the second encoder, and the first decoder, or the minimum reconstruction loss value within a part of parameter value ranges of the first encoder, the second encoder, and the first decoder, or the minimum value obtained within a preset number of iteration steps. The embodiments of the present application are not limited thereto and can be adjusted according to the actual usage environment.

[0079] In one embodiment, the preset condition is set such that the reconstruction loss is less than or does not exceed a preset value, i.e., if the reconstruction loss drops to a certain threshold, it is determined that the preset condition is met and the training step is terminated.

[0080] In this manner, S1052 and S1053 adjust the parameters of the first encoder, second encoder, and first decoder models in an iterative loop collectively until the reconstruction loss of the first sample group in the automatic codec network meets the preset condition.

[0081] It should be noted that in a practical application scenario, when the use cases of multiple information synthesis are similar, the automatic codec network is trained during the first information synthesis, and the trained automatic codec network is used continuously during subsequent synthesis. In addition, by recording the parameters of the first encoder, the second encoder, and the first decoder that have been trained for a specific type of use case, the parameters can be directly used each time the corresponding information is synthesized without the need to repeat the training. That is, in the embodiment shown in FIG. 1, the parameters of the first encoder, the second encoder, and the first decoder may be adjusted to a state that meets the usage requirements by training in advance when speech synthesis starts, so there is no need to execute S105.

[0082] The reconstruction loss is an index for determining whether the training process can be terminated during training. In one embodiment, as shown in FIG. 4, the reconstruction loss is obtained in the following manner. FIG. 4 is a diagram showing the information flow of the first decoder, the second encoder, and the training of the first encoder in an embodiment of the present application. In it, the solid arrows represent the information flow of obtaining the reconstruction loss from the first sample group and the second sample group, the dashed arrows represent the information flow of adjusting the first decoder, the second encoder, and the first encoder based on the feedback of the reconstruction loss, the solid box illustrates the parameters (scalar, vector, matrix, etc.) in the information flow, and the dashed box illustrates the arithmetic operation of the parameters (function, model, etc.).

[0083] The solid arrows in FIG. 4 indicate the process of obtaining the reconstruction loss. First, the first encoder determines a first distribution parameter based on each first sample. Among them, the first distribution parameter characterizes the first distribution of the first code, and the first code is the code of the first sample. The first distribution here is regarded as the intermediate state of the first target distribution in the training process. Similarly, it is understood that the first distribution is the posterior distribution of the first code from the first sample, and is usually a preset distribution type. In a practical application scenario, the first encoder can encode each first sample to obtain a first code representing the first sample, and then determine the first distribution parameter of the posterior distribution based on the first code of each first sample. Specifically, the first distribution parameter can be obtained based on the first code in one or more fully connected layers embedded in the first encoder. When the first distribution parameters include multiple elements (such as mean and variance), each fully connected layer is used to obtain one element therein. Typically, the first distribution is a multidimensional distribution whose dimensions correspond to the number of first samples in the first sample group (i.e., the number of second samples in the second sample group).

[0084] After obtaining the first distribution, the same steps as in the information synthesis process are performed. First, a sampling code of each first sample is obtained based on the first distribution. Specifically, a sampling code of each first sample is obtained by obtaining a first distribution parameter. For example, if the first sample group has M first samples, and each sample obtains one code element after being encoded by the first encoder, the first distribution is an M-dimensional normal distribution, and each dimension k (1≦k≦M) has a mean value μ k and variance Σ k In this case, μ k and Σ k Based on the sampling of one code element c k We can obtain {c1,c2,…c M} is a sampling code of each first sample. This example is merely for illustrating the process of obtaining the sampling code, and the present application is not limited thereto. At the same time as obtaining the sampling code of each first sample, the second encoder encodes the second sample group to obtain a second code of the second sample. Then, a second distribution parameter is obtained based on the second code. Specifically, the second distribution parameter can be obtained based on the second code in one or more fully connected layers embedded in the second encoder. When the second distribution parameter includes multiple elements (such as the mean value and the variance), each fully connected layer can obtain the second distribution parameter. The second distribution parameter represents a feature of the second distribution of the second code, and the second code of the second sample corresponds to the first code of the first sample. The second code represents a feature extracted from the second sample by the second encoder, and the second distribution is directly determined by the second sample, and is essentially a prior distribution of the second code. Then, the first decoder decodes the sampling code of the first sample according to the second code of the second sample corresponding to each first sample, to obtain a reconstructed sample of the first sample. The reconstructed samples of each first sample constitute a reconstructed sample group.

[0085] It is understood that the second distribution should be used to predict the first target distribution so that the input information in the information synthesis process effectively constrains the output information. Therefore, the second code of the second sample should correspond to the first code of the first sample. For example, when the first sample and the second sample are corresponding speech samples and text samples, respectively, the component representing an inter-sentence pause in the second code can correspond to the component representing a text clause in the first code.

[0086] Then, calculate the reconstruction loss. First, obtain a first dissimilarity between the first distribution and the second distribution based on the first distribution parameters and the second distribution parameters. It is understood that an important goal in the training process is to use the second distribution to predict the final first distribution (first target distribution). Therefore, the dissimilarity between the two is an important parameter used to evaluate the training result. The first dissimilarity is calculated by a first dissimilarity function. The calculation includes, but is not limited to, difference calculation, mean square error calculation of distribution parameters, relative entropy calculation, cross entropy calculation, etc. In the embodiment of the present application, there is no particular limitation as long as it reflects the dissimilarity between the two distributions.

[0087] Also, obtain a second dissimilarity between the first sample group and the reconstructed sample group according to the first sample group, the reconstructed sample group and the second dissimilarity function. It is understood that the important task of the first decoder is to generate a new sample similar to the first sample, and the degree of reconstruction recovery is also an important parameter for evaluating loss. The second dissimilarity is calculated according to the second dissimilarity function. The calculation includes, but is not limited to, difference calculation, mean square error calculation of distribution parameters, etc. In the embodiment of the present application, there is no particular limitation as long as it reflects the dissimilarity between two sample groups.

[0088] Finally, a reconstruction loss is obtained based on the first dissimilarity and the second dissimilarity. That is, the reconstruction loss must simultaneously consider the dissimilarity between the first distribution and the second distribution, and the dissimilarity between the first sample group and the second sample group. The reconstruction loss here is calculated by a reconstruction loss function. The calculation includes, but is not limited to, addition, multiplication, weighting, exponential operation, etc. In the embodiment of the present application, there is no particular limitation as long as the reconstruction loss has a positive correlation with both the first dissimilarity and the second dissimilarity.

[0089] In one embodiment, in order to easily calculate the first dissimilarity, the first distribution and the second distribution can be of the same type. For example, if the first distribution and the second distribution are both normal distributions, the first distribution parameters can include the expectation and variance of the first distribution, and the second distribution parameters can include the expectation and variance of the second distribution.

[0090] Next, let us refer to the dashed arrow part in FIG. 4, that is, the feedback process of the reconstruction loss. The reconstruction loss can be simultaneously fed back to the first encoder (feedback A), the second encoder (feedback B) and the first decoder (feedback C) during the feedback. Among them, feedback A, feedback B and feedback C may be the same or different. That is, the reconstruction loss is the same shape. The feedback may be given in the form of the reconstruction loss function, or in a different form (feedback of a different part of the reconstruction loss function, or feedback of a different calculation result based on the reconstruction loss). Since the training is an iterative process, each time it is determined that the reconstruction loss obtained by the calculation does not meet the preset condition, it is determined how to adjust the auto-codec network based on the currently obtained reconstruction loss. In one embodiment, the feedback is realized by gradient backpropagation of the reconstruction loss. That is, the gradient of the change in the reconstruction loss is obtained from the currently obtained reconstruction loss and the previously obtained reconstruction loss, and the gradient is fed back to the auto-codec network as a basis for adjusting the first encoder, the second encoder, and the first decoder parameters.

[0091] In the above process, the first dissimilarity is reduced by the reduction of the reconstruction loss. Since the parameters of the first encoder and the second encoder are both adjusted according to the back-propagated gradient, the reduction of the first dissimilarity is essentially a process in which the first distribution and the second distribution approach each other, i.e., the posterior distribution from the first sample group to the first feature and the prior distribution from the second sample group to the second feature approach each other. As mentioned above, information synthesis essentially aims to reconstruct output information similar to the first sample in controlling input information, so the training process in the automatic codec network aims to make the posterior distribution learn as much information as possible from the first sample group under the constraint of the prior distribution. Therefore, if the prior distribution during training strongly constrains the posterior distribution (the first distribution closely approaches the second distribution, but the second distribution does not approach the first distribution that closely), the final posterior distribution may be overly biased to the prior distribution, making it impossible to learn useful information.

[0092] To solve the above problem, in one embodiment, the reconstruction loss can be processed before gradient backpropagation. For example, the reconstruction loss is divided into a first part and a second part. The reconstruction loss of the first part is gradient cut to the second distribution (parameters of the second decoder) and the gradient is not backpropagated. That is, the gradient change of the reconstruction loss of the part does not affect the adjustment of the second decoder parameters. The reconstruction loss of the second part is gradient cut to the first distribution (parameters of the first decoder) and the gradient is not backpropagated. That is, the gradient change of the reconstruction loss of the part does not affect the adjustment of the first decoder parameters. In this case, the proportion of the first part and the second part in the reconstruction loss is controlled to control the degree to which the first distribution and the second distribution approach each other during training. Usually, the proportion of the first part in the reconstruction loss is preset to be larger than the second part, and the first distribution is prevented from approaching the second distribution so as to lose the opportunity to learn useful information from the first sample group.

[0093] After the training process is completed, i.e., after obtaining a reconstruction loss that meets the preset condition, the trained first encoder, second encoder and first decoder are obtained. Then, the information synthesis process shown in S101 to S104 above is performed. In this case, the first distribution parameters are determined based on the trained parameters of the first encoder and / or second encoder and / or the inputs corresponding to the first and / or second encoder, i.e., the final first distribution is taken as the first target distribution.

[0094] Corresponding to the training process shown in FIG. 4, FIG. 5 is a diagram showing the information flow in performing information synthesis using a trained first decoder, a second encoder, and a first encoder in an embodiment of the present application. As shown in FIG. 5, in the information synthesis process, a first target distribution is determined based on a trained automatic codec network (FIG. 5 shows the trained first encoder), and then sampling is performed based on the first target distribution to obtain a target sampling code. For example, if the first distribution is a normal distribution, the first target distribution is naturally also a normal distribution, and the target sampling code is a code obtained by sampling from the normal distribution. The target sampling code is It is understood that one "virtual" first sample is regarded as a first code obtained after passing through a first encoder, or a sampling code generated from a trained first encoder, and the related attributes thereof can be referred to in the above description of the first code and the sampling code, so they will not be described here. Also, as shown in FIG. 5, a first target distribution is obtained from the parameters of a trained first decoder, but since the first distribution and the second distribution in this case are close to each other by training so as to meet the preset conditions, it is understood that the first target distribution can be obtained from a trained second encoder, and the effect is similar. For example, text information is input to a second encoder, the second encoder obtains a text code, and the parameters obtained by the text code passing through a fully connected layer are used as the parameters of the first target distribution, and the first target distribution is obtained by fitting the parameters of the first target distribution.

[0095] In addition, the trained second decoder encodes the input information to obtain the target code of the corresponding input information. Then, the target code and the target sampling code are input to the second decoder at the same time, and the second decoder decodes the target sampling code under the condition of the first code to obtain the output information. For example, the second decoder corresponds to the decoding side of the conditional variational autodecoder, and the target code in this case can participate in the decoding process as a tag. The output information corresponds to the input information, while it is understood that it is new information similar to the first sample and has the same type as the first sample, that is, the autocodec network in this case has completed the process of information synthesis.

[0096] In order to clearly understand the above-mentioned information synthesis method, the following will take the use case of speech synthesis as an example to describe a specific embodiment of the present application. FIG. 6 is a diagram showing the information flow in a speech synthesis application scenario of the information synthesis method in the embodiment of the present application. In this embodiment, the input information and the second sample are both set as text type information, that is, the text input is Y. The output information f(z, Y') and the first sample are set as voice type information, that is, the first sample is the voice input X. The automatic speech coding network is configured to adopt Tacotron decoded frames and transducer (Transformer) encoded frames, specifically, the first encoder is set as the encoding side of the VAE encoder, and the second encoder is set as the BERT encoder, and the first decoder is the decoding side corresponding to the VAE encoder, and a CVAE structure is formed from the output of the VAE encoder and the BERT encoder. In the above structure, the VAE encoder mainly plays the role of enhancing the prosody and stability of the speech synthesis process, and the BERT encoder mainly plays the role of understanding the meaning of the text and capturing the prosody of the text sentence.

[0097] First, refer to the training process in Fig. 6. In this embodiment, when training the automatic codec network, the first and second samples with one-to-one correspondence (text-speech sample pair) are generated without the need to input phonemes and manually extracted text features to the first and second encoders.<y,x> ), where y is the input text (the second sample in the second sample group Y) and x is the input speech (the first sample in the first sample group X). Typically, the format of x is a speech waveform file (wave file) or a fundamental frequency cepstral feature extracted from a wave file.

[0098] In this embodiment, the VAE encoding side uses X to determine the first distribution parameter, and determines the posterior distribution from the first sample to the first code, which is expressed as Q(Z|X). Among them, Z represents the latent layer feature of X after passing through the VAE encoding side, that is, the first code Z=VAE encoder (X). The BERT encoder uses Y to obtain a second code Y', then fits the distribution of the second code with a predetermined distribution type to obtain second distribution parameters, and determines the prior distribution of the second code, denoted as P(Z'|Y). Z' represents the second code of Y after going through the BERT encoder, and since the second code Z' and the first code Z correspond to each other, P(Z'|Y) can be understood as the prior distribution of P(Z|Y), i.e., the first code Z.

[0099] In one embodiment, the first distribution and the second distribution are both normal distributions, and the formula TIFF0007605997000001.tif6170 It is expressed as:

[0100] In this case, the first distribution parameters include expectation μ1(X) and variance Σ1(X), where μ1(X) and Σ1(X) are typically expressed in the form of vectors with a number of elements equal to the number of first samples x in the first sample group X, i.e., each x corresponds to one variance and one expectation. Similarly, the second distribution parameters include μ2(Y) and variance Σ2(Y).

[0101] After obtaining the first distribution and the second distribution, a first dissimilarity can be obtained, which in this embodiment is set as the relative entropy of the first distribution and the second distribution, and expressed in the form of KL (Kullback-Leibler) divergence. For example, the first dissimilarity function is expressed by Equation (1).

number

[0102] Among them, D KL means to find the KL divergence. When the first distribution and the second distribution are both normal distributions, Equation (1) becomes Represented as TIFF0007605997000003.tif6170.

[0103] Then, a set of latent layer features Z corresponding to X is obtained based on N1 sampling. Y' is input to the decoding side as a tag, and the latent layer features Z are controlled to be decoded, and X reconstructed by the automatic codec network, that is, the reconstructed sample group f(Z, Y') is obtained. In the decoding process, it is understood that the reconstruction of the first sample x corresponding to each second sample y of Y' is controlled by the second code y' corresponding to the second sample y, so that the number of reconstructed samples of f(Z, Y') is consistent with the number of first samples of X. In this case, a second dissimilarity can be obtained. In this embodiment, it is set as the mean square error between the first sample group and the reconstructed sample group, for example, the second dissimilarity function is expressed by Equation (2).

number

[0104] Among them, MSE means calculating the mean square error.

[0105] Therefore, the overall reconstruction loss L of the auto-codec network is expressed as:

number

[0106] Among the above reconstruction losses, the mean square error constraint on L2 is effective for the VAE encoding side to learn information for speech synthesis (such as prosody) from the first sample group X, and the KL divergence constraint on L1 restricts the VAE encoding side from learning the synthesis information while imposing a distribution restriction. Although L is expressed as the sum of L1 and L2 in the above, it is understood that the embodiment of the present application is not limited to this. In a real application scenario, L is the sum of L1 and L It can be expressed in other forms, such as the product of L1 and L2, or the weighting of L1 and L2, as long as it has a positive correlation with both L1 and L2.

[0107] After obtaining the reconstruction loss L, gradient backpropagation is used to feed back L to the auto-codec network to adjust the first encoder, the second encoder and the first decoder. In one embodiment, to avoid over-restricting the prior distribution on the posterior distribution, L1 of the reconstruction loss L is divided into two parts to participate in gradient backpropagation.

number

[0108] Among them, α and β respectively represent the proportion of the KL divergence of the two items in L1, and except for the difference in the proportion, the magnitude of the KL divergence of the two items is the same. Among them, sg means that the gradient is cut and the gradient is not backpropagated. That is, the distribution in the parentheses after sg is not affected by the gradient change in the feedback process. Therefore, the KL divergence of the first item in the above formula represents the degree to which the posterior distribution approaches the prior distribution. Backpropagating the item to the first encoder (VAE encoding side) is for the purpose of parameter adjustment, so it is not backpropagated to the second encoder (BERT encoder). The KL divergence of the second item in the above formula represents the degree to which the prior distribution approaches the posterior distribution. Backpropagating the item to the second encoder is for the purpose of parameter adjustment, so it is not backpropagated to the first encoder. The above three items constitute a joint training model of the reconstruction loss function. It is understood that the main purpose of the second encoder in the training process is to find a rule from the first sample, and then fully utilize the rule in the synthesis process to reconstruct samples based on input information, so that the prior distribution is usually closer to the posterior distribution. In order to make the degree to which the posterior distribution approaches the prior distribution smaller than the degree to which the prior distribution approaches the posterior distribution, α<β (for example, 10α=β) is usually used.

[0109] The above training process has at least two major differences compared to existing speech synthesis methods.

[0110] First, conventional speech synthesis methods (such as Tacotron) use a simple encoding-decoding model to predict the output of a group. Typically, the decoder predicts the output of a group by using a context text C obtained after encoding on the encoder side and a sequence of historical speech frames x1, x2, …, x obtained after decoding on the decoder side. T-1 Based on the current speech frame x T predict, i.e. TIFF0007605997000007.tif10170. Among them, P θrepresents the conditional probability from one previous speech frame to the next speech frame. However, the speech data used usually contains a large amount of noise and potential pronunciation errors, which greatly increases the uncertainty of the data. In addition, it is difficult to explain such uncertainty using the corresponding text data, and the simple encoding-decoding model as described above cannot absorb these noises and pronunciation errors, which increases the difficulty of model learning. In contrast, the first encoder (such as the VAE encoding network) in the embodiment of the present application not only simulates these noises and pronunciation errors using the first distribution, but also simulates prosody, which is generally difficult to model. Therefore, when predicting a speech frame, in addition to using the prosody provided by the history sequence and context text C, these noises, pronunciation errors and prosody can be learned using the latent layer features Z provided by the first encoder, i.e., the current speech frame is TIFF0007605997000008.tif10170, the difficulty of learning the model is reduced. It is understood that when used in the field of speech recognition, other types of information may be used as the input information and the output information, in which case the above historical speech frame sequences may be replaced with corresponding historical output information, and thus the description is omitted here.

[0111] Next, in a conventional VAE model, even if the posterior distribution is assumed to be normal, the posterior distribution is usually predicted using a standard normal distribution, i.e., N(0,1) as the prior distribution. This makes the input to the decoding side zero or close to zero after predicting the posterior distribution. Although some noise is removed, the prosodic information contained in the latent layer feature Z is also lost, resulting in a synthesis result with less noise but less prosodic fluctuation. In contrast, in an embodiment of the present application, the second sample is encoded using a second encoder, and then the posterior distribution is predicted using a prior distribution obtained from the encoding result. The prior distribution includes text features corresponding to the speech data, and retains the prosodic fluctuation corresponding to the speech, thereby removing noise while avoiding the loss of prosodic information of the speech.

[0112] Next, refer to the speech synthesis process in FIG. 6. The speech synthesis part in the dashed frame in FIG. 6 is a schematic explanation of the process. When performing speech synthesis, since there is no actual x input to the first decoder (VAE decoding side), it is not possible to obtain a first distribution based on X as in the training process. In this case, the first target distribution predicted by the prior distribution is directly used for sampling to obtain a target sampling code Z (latent feature Z). The target sampling code Z is input to the decoding side together with a target code Y' generated from the code text Y of the second encoder (BERT encoder), and speech f(Z, Y') corresponding to Y is synthesized by decoding. In this case, the target sampling code Z obtained by sampling is still regarded as a latent feature of the speech signal, but since what the second encoder processes is a text input, it is understood that Z contains prosodic and noise information that is not processed by the second encoder. Therefore, for what the posterior distribution learns from the prior distribution, it is mainly prosodic information predicted from the text. These prosodic information further reduce the noise in the finally synthesized speech.

[0113] Although Fig. 6 shows a use case of speech synthesis, i.e., generating speech information using text information, the information synthesis method of the embodiment of the present application can also be applied to other use cases. For example, image information can be generated based on text information, speech information can be generated based on image information, and text information can be generated based on speech information. The first decoder, the second decoder, and the first encoder can be replaced with appropriate artificial neural network models.

[0114] The above describes a speech synthesis method including a training process in an embodiment of the present application, with reference to S105, etc. In some embodiments, a second encoder can be pre-trained before training the entire automatic codec network, so as to improve the efficiency of the training process and speech synthesis. FIG. 7 is a flowchart showing another information synthesis method in an embodiment of the present application. As shown in FIG. 7, based on the method shown in FIG. 2, the information synthesis method before S105 may include the following steps:

[0115] S106: Initialize a second encoder based on the pre-trained sample group.

[0116] In this step, the pre-trained sample group includes one or more pre-trained samples. Usually, the type of the pre-trained sample and the type of the input information are the same and the number of them is large, so that the parameters of the second encoder are initialized to better understand the input information and the second sample group, and the rationality of prediction in the automatic codec network is improved. For example, when the second encoder is a BERT model, the pre-trained sample group includes one or more pre-trained samples. The group is a large amount of text, which adds prosody to the synthesized output information (such as speech synthesis) based on a large amount of easy-to-obtain text pre-training data, and compensates for potential problems such as unreasonable divisions and unreasonable pauses in synthetic speech due to front-end deficiencies.

[0117] Next, the advantages of the information synthesis method of the embodiment of the present application will be explained in comparison with the conventional technology, taking multilingual speech synthesis as an example. Table 1 shows the evaluation results obtained by the Mean Opinion Score in the same listening test for the Google (Google Inc.) translation system and a speech synthesis system to which the information synthesis method of the embodiment of the present application is applied (hereinafter referred to as the "Base system"). The evaluation targets are the naturalness (rhythm, timbre, etc.) of speech synthesis in six minor languages: Portuguese, Hindi, Indonesian, German, Italian, and Malay. [Table 1]

[0118] The comparison in Table 1 shows that in speech synthesis, the Base system outperforms the Google Translate system in terms of both individual language MOS and overall average MOS.

[0119] Another aspect of the embodiment of the present application provides an information synthesis device. In this embodiment, the information synthesis device is used in an automatic codec network, which includes a first encoder, a second encoder and a first decoder model. Details of the automatic codec network can be referred to in the above-mentioned method embodiment, so they are omitted here.

[0120] Hereinafter, the information synthesis device will be described with reference to Fig. 8. Fig. 8 is a diagram showing the structure of the information synthesis device in the embodiment of the present application. As shown in Fig. 8, the information synthesis device includes the following modules:

[0121] Distribution acquisition module 801: obtain a first target distribution based on the trained auto-codec network, where the first target distribution represents the characteristics of the posterior distribution from the first encoder input side to the first encoder output side, during the training process of the auto-codec network, the input samples of the first encoder and the second encoder have a one-to-one correspondence, and the prior distribution determined by the second encoder is used to predict the first target distribution.

[0122] Sampling module 802: Sampling based on a first target distribution to obtain a target sampling code.

[0123] Encoding module 803: Encoding the input information through the trained second encoder to obtain a target code of the input information.

[0124] Decoding module 804: Using the trained first decoder as the decoding condition, decode the target sampling code to obtain output information.

[0125] 9 is a diagram showing the structure of another information synthesis device in an embodiment of the present application. Based on the structure shown in FIG. 8, the information synthesis device further includes the following modules:

[0126] A training module 805: Trains a first encoder, a second encoder, and a first decoder to obtain a trained first encoder, a trained second encoder, and a trained first decoder.

[0127] In one embodiment, the training module 805 includes the following units:

[0128] An input unit: inputs a first sample group and a second sample group into the automatic codec network, where the first sample group includes a first sample, the type of the first sample and the output information are consistent, the second sample group includes a second sample, the type of the second sample and the type of the input information are consistent, and the second sample and the first sample have a one-to-one correspondence.

[0129] An adjustment unit: adjusts parameters of the first encoder, the second encoder and the first decoder until the reconstruction loss for the first sample group of the auto-codec network meets a preset condition.

[0130] In one embodiment, the preset condition is that the reconstruction loss is a minimum value or that the reconstruction loss is less than a preset value.

[0131] In one embodiment, the reconstruction loss is obtained by: a first encoder determining, based on each first sample, a first distribution parameter representing a first distribution characteristic of a first code, which is a code of the first sample; sampling based on the first distribution to obtain a sampling code for each first sample; a second encoder encoding a second sample group to obtain a second code for a second sample corresponding to the first code of the first sample; and obtaining, based on the second code, a second distribution parameter representing a second distribution characteristic of the second code; The method includes: obtaining a first dissimilarity between the first distribution and the second distribution according to the distribution parameters and the second distribution parameters; a first decoder decodes the sampling code of the first sample according to the second code of the second sample corresponding to each first sample, and obtains the reconstructed samples of the first sample constituting a reconstructed sample group; obtaining a second dissimilarity between the first sample group and the reconstructed sample group according to the first sample group and the reconstructed sample group; and obtaining a reconstruction loss according to the first dissimilarity and the second dissimilarity, where the trained parameters of the first encoder are the parameters of the first encoder when the reconstruction loss satisfies a preset condition, and the first target distribution is the first distribution when the reconstruction loss satisfies a preset condition.

[0132] In one embodiment, the first dissimilarity measure is the relative entropy of the first distribution and the second distribution.

[0133] In one embodiment, a first proportion of the first dissimilarity is applied to the second encoder during gradient backpropagation. A first dissimilarity of a second ratio is backpropagated to a second encoder without backpropagating to the first encoder during gradient backpropagation, and the first ratio is smaller than the second ratio.

[0134] In one embodiment, the first distribution and the second distribution are the same type of probability distribution.

[0135] In one embodiment, the first distribution and the second distribution are normal distributions, and the first distribution parameters and the second distribution parameters both include an expectation and a variance of the normal distribution.

[0136] 10 is a diagram showing the structure of another information synthesis device in an embodiment of the present application. Based on the structure shown in FIG. 9, the information synthesis device further includes the following modules:

[0137] An initialization module 806: initializes the second encoder based on a pre-trained group of samples, each of which matches the type and input information of the sample.

[0138] In one embodiment, the first encoder is the encoding side of a conditional variational autoencoder, and the first decoder is the decoding side of the conditional variational autoencoder.

[0139] In one embodiment, the second encoder is a bidirectional encoder representation model based on a transducer.

[0140] In one embodiment, the input information is text and the output information is audio.

[0141] In one embodiment, the first decoder determines the audio information of a next frame based on a decoded historical frame sequence in the audio information and contextual text in the text information.

[0142] In one embodiment, when training the first encoder, the second encoder and the first decoder, it is not necessary to input phonemes and manually extracted text features to the first encoder and the second encoder.

[0143] According to the information synthesis device of the embodiment of the present application, a first target distribution is obtained based on the trained automatic codec network, and sampling is performed based on the first target distribution to obtain a target sampling code, a trained second encoder encodes input information and obtains a target code for the input information, and a trained first decoder uses the target code as a decoding condition to decode the target sampling code and obtains output information. Among them, the first target distribution represents the characteristics of the posterior distribution from the input side to the output side of the first encoder, and during the training process of the automatic codec network, the input samples of the first encoder and the second encoder correspond one-to-one, and the prior distribution determined by the second encoder is used to predict the first target distribution. As a result, the correspondence between the decoded output information and the input information highly restores (or simulates) the correspondence between the input samples of the first encoder and the second encoder in the training sample, so that output information that corresponds well to the input information can be obtained. The present solution alleviates limitations on the quality of speech samples in the field of speech synthesis, especially in the field of multilingual speech synthesis, by obtaining a large number of training samples, so that the final synthesized speech has better performance in areas such as prosody, pauses and character pronunciation.

[0144] Another aspect of an embodiment of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing instructions that, when executed by the processor, cause the electronic device to perform any one of the information synthesis methods described above.

[0145] Another aspect of an embodiment of the present application provides a non-transitory computer readable storage medium having instructions stored thereon that, when executed by the processor, perform any one of the information synthesis methods described above.

[0146] The computer readable storage medium includes program commands, data files, data structures, etc., or combinations thereof. The program recorded on the computer readable storage medium may be designed or configured to implement the method of the present invention. The computer readable storage medium includes a hardware system that stores and executes the program commands. The hardware system includes magnetic media (hard disks, floppy disks, magnetic tapes, etc.), optical media (CD-ROMs, DVDs, etc.), and magneto-optical media (soft optical disks, ROM, RAM, flash memory, etc.). The program includes assembly language code or machine code compiled by a compiler and higher level language code interpreted by an interpreter. The hardware system can be implemented using at least one software module to conform to the present invention.

[0147] Those skilled in the art can also recognize that the units and algorithm steps illustrated in the embodiments disclosed herein may be implemented in electronic hardware, computer software, or a combination of both. In order to clearly show the compatibility of hardware and software, the configurations and steps of each example are generally described by function in the above description. Whether these functions are performed by hardware or computer software depends on the specific application and design constraints of the technical proposal. Those skilled in the art can use different methods for each specific application to realize the described functions, but such implementation should not be considered as going beyond the scope of the present invention.

[0148] Each embodiment in this specification is described step by step, and each embodiment is described focusing on the differences from other embodiments, and the same and similar parts between the embodiments may be referred to each other. Since the apparatus disclosed in the embodiments corresponds to the method disclosed in the embodiments, the description will be simplified, and the relevant points will be referred to the description of the method embodiment.

[0149] The above description of the disclosed embodiments enables one skilled in the art to make or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Thus, the present invention is not limited to these embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. 1. A method of information synthesis for use in an automatic codec network including a first encoder, a second encoder, and a first decoder, comprising: Obtaining a first target distribution representing a feature of a posterior distribution from the first encoder input to the first encoder output based on a trained auto-codec network; obtaining a target sampling code based on the first target distribution; encoding input information with the trained second encoder to obtain a target code for the input information; The trained first decoder decodes the target sampling code based on the target code to obtain output information; Wherein, during the training process of the automatic codec network, the input samples of the first encoder and the second encoder have a one-to-one correspondence, and a prior distribution determined by the second encoder is used to predict the first target distribution; 13. An information synthesis method comprising:

2. training the first encoder, the second encoder, and the first decoder to obtain the trained first encoder, the trained second encoder, and the trained first decoder.

2. The information synthesis method according to claim 1 .

3. and prior to training the first encoder, the second encoder, and the first decoder, initializing the second encoder based on a pre-trained sample group that matches each sample type with the input information.

3. The information synthesis method according to claim 2.

4. Training the first encoder, the second encoder, and the first decoder includes: inputting a first sample group including a first sample and a second sample group including a second sample into the auto-codec network; The parameters of the first encoder, the second encoder, and the first decoder are adjusted based on a predicted reconstruction loss for the first sample group of the automatic codec network. and adjusting the temperature until it meets the conditions set for the temperature regulation. wherein the type of the first sample matches the type of the output information, the type of the second sample matches the type of the input information, and the second sample has a one-to-one correspondence with the first sample; 3. The information synthesis method according to claim 2.

5. The acquisition of the reconstruction loss includes: determining, by the first encoder, based on each of the first samples, first distribution parameters characteristic of a first distribution of a first code that is a code of the first samples; obtaining a sampling code for each of the first samples based on the first distribution; said second encoder encoding said second group of samples to obtain second codes for said second samples corresponding to said first codes for said first samples; obtaining second distribution parameters based on the second code, the second distribution parameters characterizing a second distribution of the second code; obtaining a first dissimilarity between the first distribution and the second distribution based on the first distribution parameter and the second distribution parameter; the first decoder decodes sampling codes of the first samples based on the second codes of the second samples corresponding to the first samples to obtain reconstructed samples of the first samples constituting a reconstructed sample group; obtaining a second dissimilarity measure between the first sample group and the reconstructed sample group based on the first sample group and the reconstructed sample group; obtaining the reconstruction loss based on the first dissimilarity measure and the second dissimilarity measure; Wherein, the trained first encoder parameters are the first encoder parameters when the reconstruction loss satisfies a preset condition, and the first target distribution is the first distribution when the reconstruction loss satisfies the preset condition.

5. The information synthesis method according to claim 4.

6. The preset condition is that the reconstruction loss is a minimum value or that the reconstruction loss is smaller than a preset value.

5. The information synthesis method according to claim 4.

7. In a process of adjusting parameters of the first encoder, the second encoder, and the first decoder until the reconstruction loss of the first sample group conforms to a preset condition, a portion of the first dissimilarity related to a term having a first ratio as a coefficient is backpropagated to the first encoder without being backpropagated to the second encoder during gradient backpropagation, a portion of the first dissimilarity related to a term having a second ratio as a coefficient is backpropagated to the second encoder without being backpropagated to the first encoder during gradient backpropagation, and the first ratio is smaller than the second ratio.

6. The information synthesis method according to claim 5,

8. the first dissimilarity measure is the relative entropy of the first distribution and the second distribution; 6. The information synthesis method according to claim 5,

9. the first distribution and the second distribution are the same type of probability distribution; 6. The information synthesis method according to claim 5,

10. the first encoder is an encoding side of a conditional variational autoencoder, and the first decoder is a decoding side of the conditional variational autoencoder; 10. The information synthesis method according to claim 1, wherein the information synthesis method further comprises:

11. the second encoder being a transducer-based bidirectional encoder representation model; 10. The information synthesis method according to claim 1, wherein the information synthesis method further comprises:

12. The input information is text and the output information is audio.

10. The information synthesis method according to claim 1, wherein the information synthesis method further comprises:

13. The first decoder determines audio information of a next frame based on the decoded history output information in the audio and the context text in the text.

13. The information synthesis method according to claim 12.

14. There is no need to input phonemes and manually extracted text features to the first encoder and the second encoder when training the first encoder, the second encoder and the first decoder.

13. The information synthesis method according to claim 12.

15. An information synthesis device for use in an automatic codec network including a first encoder, a second encoder, and a first decoder, comprising: A distribution acquisition module for obtaining a first target distribution representing a feature of a posterior distribution from the first encoder input side to the first encoder output side based on a trained auto-codec network; a sampling module for obtaining a target sampling code based on the first target distribution; an encoding module that encodes input information through the trained second encoder to obtain a target code for the input information; a decoding module for decoding the target sampling code based on the target code through the trained first decoder to obtain output information; Wherein, during the training process of the automatic codec network, the input samples of the first encoder and the second encoder have a one-to-one correspondence, and a prior distribution determined by the second encoder is used to predict the first target distribution; 1. An information synthesis device comprising:

16. a training module for training the first encoder, the second encoder, and the first decoder to obtain the trained first encoder, the trained second encoder, and the trained first decoder.

16. The information synthesis device according to claim 15.

17. and an initialization module for initializing the second encoder based on a pre-trained sample group in which each sample type and the input information match.

17. The information synthesis device according to claim 16.

18. The training module includes: an input unit for inputting a first sample group including a first sample and a second sample group including a second sample into the auto-codec network; The parameters of the first encoder, the second encoder, and the first decoder are adjusted based on a predicted reconstruction loss for the first sample group of the automatic codec network. and an adjustment unit for adjusting the temperature until the temperature meets the set conditions; wherein the type of the first sample matches the type of the output information, the type of the second sample matches the type of the input information, and the second sample has a one-to-one correspondence with the first sample; 17. The information synthesis device according to claim 16.

19. An electronic device including a memory and a processor, The memory stores instructions which, when executed by the processor, perform the information synthesis method of any one of claims 1 to 14.

1. An electronic device comprising:

20. 1. A non-transitory computer-readable storage medium, comprising: The computer-readable storage medium stores instructions that, when executed by a processor, perform the information synthesis method according to any one of claims 1 to 14. A non-transitory computer-readable storage medium comprising:

Images