Model training method, audio generation method, electronic equipment and storage medium

Abstract

The invention discloses a model training method, an audio generation method, electronic equipment and a storage medium. The method comprises the following steps: acquiring a group of training data from a current training data set, the training data comprising source audio data and corresponding text data, and the source audio data comprising original acoustic features; inputting the original acoustic features and the text data into a generative network to generate target audio data; inputting the target audio data and the source audio data into an identification network, and determining a loss function; if the training end condition is met, taking the generation network as a sub-model, matched with the current sound source, of the audio generation model; and if the training end condition is not met, adjusting parameters of the generation network and the identification network according to the loss function, obtaining a group of training data from the current training data set again, and returning to execute the step of inputting the original acoustic features and the text data into the generation network to generate the target audio data. According to the scheme, audios with multiple timbres close to natural languages can be generated.

Description

Technical Field

[0001] The present invention relates to the field of audio processing technology, and in particular to a model training method, an audio generation method, an electronic device and a storage medium. Background Art

[0002] Text to Speech (TTS) technology is an artificial intelligence technology that converts text information into spoken output. It is widely used in voice assistants, audio e-books, navigation systems, telephone customer service, speech synthesis applications and other fields.

[0003] Existing TTS technology is usually implemented through two models: one is the text conversion model and the other is the audio synthesis model. The text conversion model mainly converts text into an intermediate representation (such as mel spectrum), and the audio synthesis model mainly converts the intermediate representation into the final speech waveform. However, the existing TTS technology can only synthesize speech with a fixed timbre, and there is a problem of insufficient feature processing in the speech synthesis process, resulting in the lack of natural fluency in the synthesized speech. Summary of the invention

[0004] The present invention provides a model training method, an audio generation method, an electronic device and a storage medium. The model only needs to be trained once to generate audio with a variety of timbres and close to natural language, thereby improving the synthesis efficiency and quality of the audio.

[0005] According to one aspect of the present invention, a model training method is provided, comprising: obtaining a set of training data from a current training data set, wherein the current training data set is a training data set corresponding to a current sound source, the training data includes source audio data and its corresponding text data, and the source audio data includes original acoustic features; inputting the original acoustic features and text data into a generation network to generate target audio data; inputting the target audio data and the source audio data into a discrimination network to determine a loss function; if a training end condition is met, the generation network is used as a sub-model of an audio generation model that matches the current sound source, wherein the sub-model is used to generate audio data that simulates the current sound source; if the training end condition is not met, adjusting the parameters of the generation network and the discrimination network according to the loss function, and re-obtaining a set of training data from the current training data set, and returning to execute the step of inputting the original acoustic features and text data into the generation network to generate the target audio data.

[0006] Optionally, the generation network includes a vector quantization layer and a fully connected layer; the original acoustic features and text data are input into the generation network to generate target audio data, including: extracting text features from the text data; inputting the original acoustic features into the vector quantization layer to obtain target acoustic features, wherein the vector quantization layer is a residual vector quantization layer, or an additive vector quantization layer, or a product vector quantization layer; inputting the target acoustic features and text features into the fully connected layer to generate target audio data.

[0007] Optionally, when the vector quantization layer is a residual vector quantization layer, the residual vector quantization layer includes several quantizers connected in sequence, and one quantizer corresponds to a coding dictionary; the original acoustic feature is input into the vector quantization layer to obtain the target acoustic feature, including: determining the input vector of the current quantizer, wherein, if the current quantizer is the first quantizer of the residual vector quantization layer, the input vector of the current quantizer is the original acoustic feature; if the current quantizer is not the first quantizer of the residual vector quantization layer, the input vector of the current quantizer is the output vector of the previous quantizer of the current quantizer, and the output vector of the previous quantizer is determined based on the input vector of the previous quantizer and the codeword of the previous quantizer; determining the codeword of the current quantizer based on the input vector of the current quantizer and the coding dictionary corresponding to the current quantizer; determining the target acoustic feature based on the codewords of all quantizers.

[0008] Optionally, after determining the codeword of the current quantizer, the method further includes: updating the coding dictionary corresponding to the current quantizer according to a preset algorithm, wherein the preset algorithm is a K-means clustering algorithm and / or a random optimization algorithm.

[0009] Optionally, the number of quantizers included in the residual vector quantization layer is 8.

[0010] Optionally, the target audio data and the source audio data are input into an identification network to determine a loss function, including: inputting the target audio data and the source audio data into an identification network to obtain first identification information and second identification information, wherein the first identification information is used to characterize the feature with the largest difference between the target audio data and the source audio data, and the second identification information is used to characterize the mapping of features between the target audio data and the source audio data except for the feature with the largest difference; constructing a loss function based on the first identification information and the second identification information.

[0011] Optionally, after using the generative network as a sub-model of the audio generation model that matches the current sound source, it also includes: obtaining a set of verification data, wherein the verification data includes verification audio data and its corresponding verification text data; inputting the verification text data into the sub-model to generate simulated audio data, and calculating the difference parameter between the verification audio data and the simulated audio data; if the difference parameter is less than or equal to a preset threshold, determining that the sub-model has passed the verification; if the difference parameter is greater than the preset threshold, after fine-tuning the parameters of the sub-model, re-obtaining a set of verification data, and returning to execute the step of inputting the verification text data into the sub-model to generate simulated audio data.

[0012] According to another aspect of the present invention, there is provided an audio generation method, comprising: determining a target sub-model matching the target sound source from an audio generation model according to a target sound source selected by a user, wherein the audio generation model comprises a plurality of sub-models, one sound source matches one sub-model, and the audio generation model is obtained using the model training method of any embodiment of the present invention; acquiring target text data, and inputting the target text data into the target sub-model to generate audio data simulating the target sound source.

[0013] According to another aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the model training method of any embodiment of the present invention, or can execute the audio generation method of any embodiment of the present invention.

[0014] According to another aspect of the present invention, a computer-readable storage medium is provided, which stores computer instructions, and the computer instructions are used to enable a processor to implement the model training method of any embodiment of the present invention when executed, or to be able to execute the audio generation method of any embodiment of the present invention.

[0015] The technical solution of the embodiment of the present invention obtains a set of training data from the current training data set, and inputs the original acoustic features of the source audio data included in the training data and the text data corresponding to the source audio data into the generation network to generate target audio data, and then inputs the target audio data and the source audio data into the identification network, determines the loss function, and adjusts the parameters of the generation network and the identification network based on the loss function, continuously trains the generation network and the identification network, and finally uses the trained generation network as a sub-model of the audio generation model that matches the current sound source to generate audio data that simulates the current sound source. The model training method, on the one hand, uses the idea of ​​generating adversarial networks to train the generation network and the identification network, so that the output result of the generation network is close to the real source audio data. Compared with the traditional method of realizing TTS technology through two models, the present invention only needs to train the model once to generate audio close to natural language, which improves the training efficiency of the model. On the other hand, since the audio generation model includes several sub-models, one sound source matches one sub-model, the audio generation model can adapt to various sound sources, providing users with more sound source options, and improving the synthesis efficiency and quality of audio.

[0016] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 It is a flowchart of a model training method provided in Embodiment 1 of the present invention;

[0019] Figure 2 This is an architecture diagram of a generation network and an identification network provided in the first embodiment of the present invention;

[0020] Figure 3 It is a flowchart of a model training method provided in Embodiment 2 of the present invention;

[0021] Figure 4 is a flowchart of an audio generation method provided in Embodiment 3 of the present invention;

[0022] Figure 5 is a structural schematic diagram of a model training device provided by Embodiment 4 of the present invention;

[0023] Figure 6 is a structural schematic diagram of an audio generating device provided by Embodiment 5 of the present invention;

[0024] Figure 7 It is a structural schematic diagram of an electronic device provided in Embodiment 6 of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "source", "target", "verification", "simulation", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] Embodiment 1

[0028] Figure 1 This is a flow chart of a model training method provided in the first embodiment of the present invention. This embodiment can be applied to the scenario of training a model for generating audio data that can simulate any sound source. The method can be executed by a model training device, which can be implemented in the form of hardware and / or software. The model training device can be configured in an electronic device (such as a computer device, a server, etc.). Figure 1 As shown, the method includes:

[0029] S110. Obtain a set of training data from a current training data set, wherein the current training data set is a training data set corresponding to a current sound source, the training data includes source audio data and its corresponding text data, and the source audio data includes original acoustic features.

[0030] In the present invention, a training data set is a data set used for model training. There are multiple training data sets, one training data set corresponds to one sound source, and the current training data set is the training data set corresponding to the current sound source. That is to say, when training the audio generation model of the present invention, each training data set needs to be used as the current training data set, and the model training method provided by the embodiment of the present invention is executed once. For example, the number of training data sets is 3, which are respectively recorded as training data set 1, training data set 2 and training data set 3. Training data set 1 corresponds to sound source 1, training data set 2 corresponds to sound source 2, and training data set 3 corresponds to sound source 3. The electronic device uses these three training data sets as the current training data sets, and executes the model training method provided by the embodiment of the present invention three times in a loop, and can obtain an audio generation model including three sub-models, and the audio generation model can generate audio data simulating these three sound sources.

[0031] In addition, the number of training data sets is variable. For example, some training data sets can be deleted or added according to actual needs. When deleting some training data sets, if the sub-models for sound source matching corresponding to the deleted training data sets have been trained, these sub-models can be deleted or retained; when adding training data sets, the sub-models for sound source matching corresponding to the added training data sets can also be continued to be trained for the added training data sets.

[0032] In one embodiment, the current training data set includes several groups of training data, each group of training data includes source audio data and its corresponding text data, and the source audio data includes original acoustic features. The original acoustic features are acoustic features extracted from the source audio data, reflecting the characteristics of the source audio data in multiple dimensions (such as pitch, timbre, rhythm, intensity, emotional color, etc.).

[0033] S120, inputting the original acoustic features and text data into the generation network to generate target audio data.

[0034] S130, inputting the target audio data and the source audio data into the identification network, and determining the loss function.

[0035] Combined with S120 and S130, the generative network and the discriminative network together constitute a generative adversarial network (GAN). The generative network can also be called a generator, and the discriminative network can also be called a discriminator or a discriminator.

[0036] Figure 2 is an architecture diagram of a generation network and an identification network provided in the first embodiment of the present invention. Figure 2As shown in the dashed box (a) in the figure, the architecture diagram of the identification network is as follows Figure 2 The generated network includes a vector quantization layer and a fully connected layer. The input and output of the fully connected layer can also be connected to the encoder and decoder. The target audio data can be generated by inputting the original acoustic features and text data into the generated network. The discriminator network includes a multi-scale discriminator. The target audio data and the source audio data are input into the discriminator network to obtain the discriminator information, which is used to determine the loss function.

[0037] In one embodiment, the vector quantization layer is a residual vector quantization layer, or an additive vector quantization layer, or a product vector quantization layer. Residual vector quantization is a multi-stage quantization method, which approximates the original vector by quantizing the original vector as a series of residual values. Its advantage is that the variance or energy of the residual is usually smaller than that of the original data, so the quantization process can be performed more efficiently. Additive vector quantization is similar to residual vector quantization, but additive vector quantization uses a complex beam search algorithm to find the closest codeword for the quantization process. Product vector quantization is a technology that decomposes the original high-dimensional vector space into the Cartesian product of multiple low-dimensional subspaces and independently quantizes each low-dimensional subspace. Its advantage is that it can generate a large codebook set from several small codebooks, thereby reducing storage requirements while maintaining adaptability to data distribution. The present invention can select the specific structure of the vector quantization layer according to actual needs.

[0038] S140: Determine whether the training end condition is met. If yes, execute S150; if no, execute S160.

[0039] In one embodiment, the training end condition includes at least one of the following three conditions:

[0040] Condition 1: The value of the loss function is less than or equal to the preset threshold;

[0041] Condition 2: The loss function converges;

[0042] Condition 3: The current number of training times is equal to the maximum number of training times.

[0043] When the training end condition includes at least two of the above conditions, as long as any one of the conditions is met, it is confirmed that the model training meets the training end condition.

[0044] S150. Using the generation network as a sub-model of the audio generation model that matches the current sound source, wherein the sub-model is used to generate audio data that simulates the current sound source.

[0045] When the training end condition is met, it means that the current generation network has been trained. At this time, the generation network can be used as a sub-model of the audio generation model that matches the current sound source. The sub-model is used to generate audio data that simulates the current sound source.

[0046] Optionally, after using the generative network as a sub-model of the audio generation model that matches the current sound source, the electronic device can also use an open source artificial intelligence (AI) model and / or language model to fine-tune the parameters of the sub-model to further improve the audio synthesis effect of the sub-model.

[0047] S160. According to the loss function, adjust the parameters of the generation network and the identification network, reacquire a set of training data from the current training data set, and return to execute S120.

[0048] When the training end condition is not met, it means that the current generation network has not been trained. At this time, according to the loss function, the parameters of the generation network and the identification network are adjusted, and a set of training data is reacquired from the current training data set, and the process returns to execute S120 until the training end condition is met.

[0049] Embodiment 2

[0050] Figure 3 is a flow chart of a model training method provided by the second embodiment of the present invention. Based on the above-mentioned first embodiment, this embodiment provides a method for optimizing the original acoustic features by a vector quantization layer, and a method for verifying the trained sub-model. Figure 3 As shown, the method includes:

[0051] S301. Obtain a set of training data from a current training data set, wherein the current training data set is a training data set corresponding to a current sound source, the training data includes source audio data and its corresponding text data, and the source audio data includes original acoustic features.

[0052] The training data set is a data set used for model training. There are multiple training data sets, one training data set corresponds to one sound source, and the current training data set is the training data set corresponding to the current sound source. That is to say, when training the audio generation model of the present invention, each training data set needs to be used as the current training data set to execute the model training method provided by the embodiment of the present invention once.

[0053] In one embodiment, the current training data set includes several groups of training data, each group of training data includes source audio data and its corresponding text data, and the source audio data includes original acoustic features. The original acoustic features are acoustic features extracted from the source audio data, reflecting the characteristics of the source audio data in multiple dimensions (such as pitch, timbre, rhythm, intensity, emotional color, etc.).

[0054] Optionally, the text data in a set of training data can be a long text or a short text after being segmented according to punctuation marks. When the text data is a short text after being segmented according to punctuation marks, the corresponding source audio data is also a short audio obtained by slicing. In this way, the amount of calculation when using each training data for training can be reduced, which facilitates parallel processing of electronic devices and improves training efficiency.

[0055] S302: Extract text features from text data.

[0056] The electronic device can use text processing tools (such as open source language models, text analysis software, etc.) to extract text features from text data. Specifically, the text features may include at least one of the following: grammatical structure, semantics, critical period, and emotional color to be expressed.

[0057] S303: Input the original acoustic features into a vector quantization layer to obtain target acoustic features, wherein the vector quantization layer is a residual vector quantization layer, an additive vector quantization layer, or a product vector quantization layer.

[0058] By quantizing the original acoustic features through the vector quantization layer, continuous audio features can be converted into discrete values, thereby reducing the size of the data, making it easier to learn and process the target acoustic features later.

[0059] In one possible implementation, the residual vector quantization layer can quantize the input vector x by applying M consecutive quantizers to it. The additive vector quantization layer works similarly to the residual vector quantization layer, but the additive vector quantization layer uses a complex beam search algorithm to find the closest codeword for the quantization process.

[0060] In another possible implementation, the product vector quantization layer can split the input vector x of dimension D into M independent subspaces of dimension D / M. Then, it applies M independent vector quantization modules to the existing subspaces. Finally, the product vector quantization quantizes the input vector x into a concatenation of M closest codewords (one for each codebook).

[0061] Exemplarily, taking the vector quantization layer as a residual vector quantization layer as an example, the method of "inputting the original acoustic feature into the vector quantization layer to obtain the target acoustic feature" in S303 is described in detail. The method includes the following three steps:

[0062] Step 1: Determine the input vector of the current quantizer, where if the current quantizer is the first quantizer of the residual vector quantization layer, the input vector of the current quantizer is the original acoustic feature; if the current quantizer is not the first quantizer of the residual vector quantization layer, the input vector of the current quantizer is the output vector of the previous quantizer of the current quantizer, and the output vector of the previous quantizer is determined based on the input vector of the previous quantizer and the codeword of the previous quantizer.

[0063] The residual vector quantization layer includes several quantizers connected in sequence, one quantizer corresponds to one coding dictionary. A coding dictionary includes multiple codewords, each of which is a vector of fixed length, representing different states or modes that may appear in the quantization process.

[0064] The specific number of quantizers can be set according to actual conditions. It is understandable that the more the number of quantizers, the more accurate the quantization result (ie, the target acoustic feature); and the fewer the number of quantizers, the faster the quantization speed.

[0065] Optionally, the number of quantizers may be 3, 4, 6, 8, or 10. Preferably, the number of quantizers is 8.

[0066] For each quantizer (i.e., the quantizer is regarded as the current quantizer), the input vector of the current quantizer is first determined. Specifically, if the current quantizer is the first quantizer of the residual vector quantization layer, the input vector of the current quantizer is the original acoustic feature; if the current quantizer is not the first quantizer of the residual vector quantization layer, the input vector of the current quantizer is the output vector of the previous quantizer of the current quantizer, and the output vector of the previous quantizer is determined based on the input vector of the previous quantizer and the codeword of the previous quantizer. Among them, the output vector of the previous quantizer is equal to the remainder of the input vector of the previous quantizer and the codeword of the previous quantizer.

[0067] Step 2: Determine the codeword of the current quantizer according to the input vector of the current quantizer and the coding dictionary corresponding to the current quantizer.

[0068] The codeword of the current quantizer is the codeword in the coding dictionary corresponding to the current quantizer that is closest to the input vector of the current quantizer.

[0069] Step 3: Determine the target acoustic features based on the codewords of all quantizers.

[0070] The target acoustic feature is equal to the sum of the codewords of all quantizers.

[0071] Exemplarily, taking the number of quantizers as 8 as an example, the method for determining the target acoustic feature is illustrated as follows: the original acoustic feature is input into the first quantizer of the residual vector quantization layer, the code word 1 closest to the original acoustic feature is searched in the coding dictionary corresponding to the first quantizer, the remainder 1 between the original acoustic feature and code word 1 is calculated, and the remainder 1 is used as the output of the first quantizer and input into the second quantizer of the residual vector quantization layer. Then, the code word 2 closest to remainder 1 is searched in the coding dictionary corresponding to the second quantizer, the remainder 2 between remainder 1 and code word 2 is calculated, and the remainder 2 is used as the output of the second quantizer and input into the third quantizer of the residual vector quantization layer. And so on, until code words 1 to code words 8 are obtained. The sum of code words 1 to code words 8 is used as the target acoustic feature.

[0072] Optionally, after determining the codeword of the current quantizer, the method further includes: updating the coding dictionary corresponding to the current quantizer according to a preset algorithm, wherein the preset algorithm is a K-means clustering algorithm and / or a random optimization algorithm.

[0073] K-means clustering algorithm (K-means) is a classic unsupervised learning algorithm used to divide samples in a data set into K clusters, where each cluster is represented by a cluster center (or centroid). The goal of the algorithm is to minimize the sum of the squares of the distances from each sample to its nearest cluster center, that is, the total inner sum of squares (SSE). Stochastic optimization algorithms are a class of algorithms that use randomization techniques in optimization problems, especially when the search space is large or the problem is difficult to solve with deterministic methods. These algorithms explore the search space by randomly selecting candidate solutions and adjust the search strategy based on feedback from the objective function.

[0074] S304: Input the target acoustic features and text features into the fully connected layer to generate target audio data.

[0075] The input and output of the fully connected layer can also be connected to the encoder and decoder. Inputting the target acoustic features and text features into the fully connected layer can combine prior coding, posterior coding, Mel spectrum coding and other means to generate target audio data.

[0076] S305. Input the target audio data and the source audio data into an identification network to obtain first identification information and second identification information, wherein the first identification information is used to characterize the feature with the greatest difference between the target audio data and the source audio data, and the second identification information is used to characterize the mapping of features between the target audio data and the source audio data except for the feature with the greatest difference.

[0077] The discriminator network includes a multi-scale discriminator, which consists of multiple sub-discriminators that work at different scales to capture different levels of features from details to the global level. This design enables the discriminator network to analyze the audio more comprehensively, thereby improving the ability to judge the authenticity of the target audio data.

[0078] In the present invention, the identification network can output two identification information: first identification information and second identification information, wherein the first identification information is used to characterize the feature with the largest difference between the target audio data and the source audio data, and the second identification information is used to characterize the mapping of features other than the feature with the largest difference between the target audio data and the source audio data. In this way, the feature with the largest difference between the target audio data and the source audio data can be isolated, which is more convenient for subsequent network optimization.

[0079] S306: Construct a loss function according to the first identification information and the second identification information.

[0080] In one embodiment, the number of loss functions may be one or two. When the number of loss functions is one, the loss function is used to guide the parameter optimization of the generation network and the identification network at the same time; when the number of loss functions is two, one loss function is used to guide the parameter optimization of the generation network, and the other loss function is used to guide the parameter optimization of the identification network.

[0081] S307: Determine whether the training end condition is met. If yes, execute S309; ​​if no, execute S308.

[0082] In one embodiment, the training end condition includes at least one of the following three conditions:

[0083] Condition 1: The value of the loss function is less than or equal to the preset threshold;

[0084] Condition 2: The loss function converges;

[0085] Condition 3: The current number of training times is equal to the maximum number of training times.

[0086] When the training end condition includes at least two of the above conditions, as long as any one of the conditions is met, it is confirmed that the model training meets the training end condition.

[0087] S308. According to the loss function, adjust the parameters of the generation network and the identification network, and reacquire a set of training data from the current training data set, and return to execute S302.

[0088] When the training end condition is not met, it means that the current generation network has not been trained. At this time, according to the loss function, the parameters of the generation network and the identification network are adjusted, and a set of training data is reacquired from the current training data set, and the execution of S302 is returned until the training end condition is met.

[0089] S309: Use the generation network as a sub-model of the audio generation model that matches the current sound source, wherein the sub-model is used to generate audio data that simulates the current sound source.

[0090] When the training end condition is met, it means that the current generation network has been trained. At this time, the generation network can be used as a sub-model of the audio generation model that matches the current sound source. The sub-model is used to generate audio data that simulates the current sound source.

[0091] Optionally, after S309, the electronic device may also fine-tune the parameters of the sub-model to further improve the audio synthesis effect of the sub-model. Specifically, as shown in S310-S314:

[0092] S310. Obtain a set of verification data, wherein the verification data includes verification audio data and its corresponding verification text data.

[0093] S311. Input the verification text data into the sub-model, generate simulated audio data, and calculate the difference parameter between the verification audio data and the simulated audio data.

[0094] In one embodiment, the difference parameter may be any one of the following: signal-to-noise ratio (SNR), total harmonic distortion (THD), dynamic range (DR), frequency response (Frequency Response), intermodulation distortion (IMD), waveform comparison (Waveform Comparison), and Mel-spectrum comparison (Mel-spectrum Comparison).

[0095] S312: Determine whether the difference parameter is less than or equal to a preset threshold value. If yes, execute S313; if no, execute S314.

[0096] S313: Determine whether the sub-model passes the verification.

[0097] S314. After fine-tuning the parameters of the sub-model, a set of verification data is acquired again, and the process returns to execute S311.

[0098] The technical solution of the embodiment of the present invention obtains a set of training data from the current training data set, and inputs the original acoustic features of the source audio data included in the training data and the text data corresponding to the source audio data into the generation network to generate target audio data, and then inputs the target audio data and the source audio data into the identification network, determines the loss function, and adjusts the parameters of the generation network and the identification network based on the loss function, continuously trains the generation network and the identification network, and finally uses the trained generation network as a sub-model of the audio generation model that matches the current sound source to generate audio data that simulates the current sound source. The model training method, on the one hand, uses the idea of ​​generating adversarial networks to train the generation network and the identification network, so that the output result of the generation network is close to the real source audio data. Compared with the traditional method of realizing TTS technology through two models, the present invention only needs to train the model once to generate audio close to natural language, which improves the training efficiency of the model. On the other hand, since the audio generation model includes several sub-models, one sound source matches one sub-model, the audio generation model can adapt to various sound sources, providing users with more sound source options, and improving the synthesis efficiency and quality of audio.

[0099] Embodiment 3

[0100] Figure 4 This is a flowchart of an audio generation method provided by Embodiment 3 of the present invention. This embodiment is applicable to the scenario of synthesizing audio using the audio generation model obtained by Embodiment 1 / Embodiment 2 above. The method can be executed by an audio generation device, which can be implemented in the form of hardware and / or software, and can be configured in an electronic device (such as a computer device, a server, etc.). Figure 4 As shown, the method includes:

[0101] S410: According to the target sound source selected by the user, determine a target sub-model matching the target sound source from the audio generation model.

[0102] The audio generation model includes several sub-models, one sound source matches one sub-model, and the audio generation model is obtained by using the model training method described in Example 1 / Example 2.

[0103] Exemplarily, it is assumed that the audio generation model includes three sub-models, which are respectively recorded as sub-model 1, sub-model 2 and sub-model 3, sub-model 1 matches sound source 1, sub-model 2 matches sound source 2, and sub-model 3 matches sound source 3. Assuming that the target sound source selected by the user is sound source 1, the electronic device can determine sub-model 1 from the audio generation model for subsequent audio synthesis according to the user's selection; assuming that the target sound source selected by the user is sound source 2, the electronic device can determine sub-model 2 from the audio generation model for subsequent audio synthesis according to the user's selection; assuming that the target sound source selected by the user is sound source 3, the electronic device can determine sub-model 3 from the audio generation model for subsequent audio synthesis according to the user's selection.

[0104] S420: Obtain target text data, and input the target text data into a target sub-model to generate audio data simulating a target sound source.

[0105] The audio generation method provided by the embodiment of the present invention can adapt to various sound sources and generate natural language audio that is close to the sound source, thereby providing users with more sound source options and improving the synthesis efficiency and quality of the audio.

[0106] Embodiment 4

[0107] Figure 5 Schematic diagram of the structure of a model training device provided by the fourth embodiment of the present invention. Figure 5 As shown, the device includes: a data acquisition module 501, a first training module 502, a second training module 503, and a model processing module 504.

[0108] The data acquisition module 501 is used to acquire a set of training data from a current training data set, wherein the current training data set is a training data set corresponding to a current sound source, the training data includes source audio data and its corresponding text data, and the source audio data includes original acoustic features;

[0109] A first training module 502 is used to input the original acoustic features and text data into a generation network to generate target audio data;

[0110] A second training module 503 is used to input the target audio data and the source audio data into the identification network and determine the loss function;

[0111] The model processing module 504 is used to, if the training end condition is met, use the generation network as a sub-model of the audio generation model that matches the current sound source, wherein the sub-model is used to generate audio data that simulates the current sound source; if the training end condition is not met, adjust the parameters of the generation network and the identification network according to the loss function, and notify the data acquisition module 501 to re-acquire a set of training data from the current training data set, so that the first training module 502 returns to execute the step of inputting the original acoustic features and text data into the generation network to generate the target audio data.

[0112] Optionally, the generation network includes a vector quantization layer and a fully connected layer; the first training module 502 is specifically used to extract text features from text data; the original acoustic features are input into the vector quantization layer to obtain target acoustic features, wherein the vector quantization layer is a residual vector quantization layer, or an additive vector quantization layer, or a product vector quantization layer; the target acoustic features and text features are input into the fully connected layer to generate target audio data.

[0113] Optionally, when the vector quantization layer is a residual vector quantization layer, the residual vector quantization layer includes several quantizers connected in sequence, and one quantizer corresponds to a coding dictionary; the first training module 502 is specifically used to determine the input vector of the current quantizer, wherein, if the current quantizer is the first quantizer of the residual vector quantization layer, the input vector of the current quantizer is the original acoustic feature; if the current quantizer is not the first quantizer of the residual vector quantization layer, the input vector of the current quantizer is the output vector of the previous quantizer of the current quantizer, and the output vector of the previous quantizer is determined based on the input vector of the previous quantizer and the codeword of the previous quantizer; determine the codeword of the current quantizer based on the input vector of the current quantizer and the coding dictionary corresponding to the current quantizer; determine the target acoustic feature based on the codewords of all quantizers.

[0114] Optionally, the first training module 502 is further used to update the coding dictionary corresponding to the current quantizer according to a preset algorithm after determining the codeword of the current quantizer, wherein the preset algorithm is a K-means clustering algorithm and / or a random optimization algorithm.

[0115] Optionally, the number of quantizers included in the residual vector quantization layer is 8.

[0116] Optionally, the second training module 503 is specifically used to input the target audio data and the source audio data into the identification network to obtain first identification information and second identification information, wherein the first identification information is used to characterize the feature with the largest difference between the target audio data and the source audio data, and the second identification information is used to characterize the mapping of features between the target audio data and the source audio data except for the feature with the largest difference; and construct a loss function based on the first identification information and the second identification information.

[0117] Optionally, the data acquisition module 501 is further used to acquire a set of verification data, wherein the verification data includes verification audio data and its corresponding verification text data;

[0118] The first training module 502 is further used to input the verification text data into the sub-model, generate simulated audio data, and calculate the difference parameter between the verification audio data and the simulated audio data;

[0119] The model processing module 504 is also used to determine that the sub-model has passed the verification if the difference parameter is less than or equal to a preset threshold; if the difference parameter is greater than the preset threshold, then after fine-tuning the parameters of the sub-model, notify the data acquisition module 501 to re-acquire a set of verification data, so that the first training module 502 returns to execute the step of inputting the verification text data into the sub-model to generate simulated audio data.

[0120] The model training device provided in the embodiment of the present invention can execute the model training method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0121] Embodiment 5

[0122] Figure 6 Schematic diagram of the structure of an audio generating device provided by Embodiment 5 of the present invention. Figure 6 As shown, the device includes: a sub-model determination module 601 and an audio generation module 602.

[0123] A sub-model determination module 601 is used to determine a target sub-model matching the target sound source from the audio generation model according to the target sound source selected by the user, wherein the audio generation model includes a plurality of sub-models, one sound source matches one sub-model, and the audio generation model is obtained by the model training method described in Embodiment 1 / Embodiment 2;

[0124] The audio generation module 602 is used to obtain target text data and input the target text data into the target sub-model to generate audio data simulating the target sound source.

[0125] The audio generating device provided in the embodiment of the present invention can execute the audio generating method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0126] Embodiment 6

[0127] Figure 7: is a structural diagram of an electronic device provided in Embodiment 6 of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0128] like Figure 7 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0129] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0130] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 performs the various methods and processes described above, such as a model training method and / or an audio generation method.

[0131] In some embodiments, the model training method and / or the audio generation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the model training method and / or the audio generation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the model training method and / or the audio generation method in any other suitable manner (e.g., by means of firmware).

[0132] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0133] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0134] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0135] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0136] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0137] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0138] An embodiment of the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the model training method and / or audio generation method provided in any embodiment of the present invention.

[0139] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0140] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0141] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A model training method, characterized in that: include: Acquire a set of training data from a current training data set, wherein the current training data set is a training data set corresponding to a current sound source, the training data includes source audio data and its corresponding text data, and the source audio data includes original acoustic features; Inputting the original acoustic features and the text data into a generation network to generate target audio data; Inputting the target audio data and the source audio data into a discrimination network and determining a loss function; If the training end condition is met, the generation network is used as a sub-model of the audio generation model that matches the current sound source, wherein the sub-model is used to generate audio data that simulates the current sound source; If the training end condition is not met, the parameters of the generation network and the identification network are adjusted according to the loss function, and a set of training data is reacquired from the current training data set, and the step of inputting the original acoustic features and the text data into the generation network to generate the target audio data is returned.

2. The model training method according to claim 1, characterized in that: The generation network includes a vector quantization layer and a fully connected layer; The step of inputting the original acoustic features and the text data into a generation network to generate target audio data comprises: Extracting text features from the text data; Inputting the original acoustic features into the vector quantization layer to obtain target acoustic features, wherein the vector quantization layer is a residual vector quantization layer, an addition vector quantization layer, or a product vector quantization layer; The target acoustic features and the text features are input into the fully connected layer to generate the target audio data.

3. The model training method according to claim 2, characterized in that: When the vector quantization layer is a residual vector quantization layer, the residual vector quantization layer includes a plurality of quantizers connected in sequence, and one quantizer corresponds to one coding dictionary; The inputting the original acoustic features into the vector quantization layer to obtain target acoustic features includes: Determine an input vector of a current quantizer, wherein if the current quantizer is the first quantizer of the residual vector quantization layer, the input vector of the current quantizer is the original acoustic feature; if the current quantizer is not the first quantizer of the residual vector quantization layer, the input vector of the current quantizer is the output vector of a previous quantizer of the current quantizer, and the output vector of the previous quantizer is determined according to the input vector of the previous quantizer and the codeword of the previous quantizer; Determine a codeword of the current quantizer according to an input vector of the current quantizer and a coding dictionary corresponding to the current quantizer; The target acoustic feature is determined according to the code words of all quantizers.

4. The model training method according to claim 3, characterized in that: After determining the codeword of the current quantizer, further comprising: According to a preset algorithm, the coding dictionary corresponding to the current quantizer is updated, wherein the preset algorithm is a K-means clustering algorithm and / or a random optimization algorithm.

5. The model training method according to claim 3, characterized in that: The residual vector quantization layer includes 8 quantizers.

6. The model training method according to claim 1, characterized in that: The step of inputting the target audio data and the source audio data into a discrimination network and determining a loss function comprises: Input the target audio data and the source audio data into an identification network to obtain first identification information and second identification information, wherein the first identification information is used to characterize the feature with the largest difference between the target audio data and the source audio data, and the second identification information is used to characterize the mapping of features other than the feature with the largest difference between the target audio data and the source audio data; The loss function is constructed according to the first identification information and the second identification information.

7. The model training method according to claim 1, characterized in that: After the generation network is used as a sub-model of the audio generation model that matches the current sound source, the method further includes: Acquire a set of verification data, wherein the verification data includes verification audio data and its corresponding verification text data; Input the verification text data into the sub-model, generate simulated audio data, and calculate the difference parameter between the verification audio data and the simulated audio data; If the difference parameter is less than or equal to a preset threshold, it is determined that the sub-model passes the verification; If the difference parameter is greater than a preset threshold, after fine-tuning the parameters of the sub-model, a set of verification data is reacquired, and the process returns to the step of inputting the verification text data into the sub-model to generate simulated audio data.

8. An audio generation method, characterized in that: include: According to the target sound source selected by the user, determining a target sub-model matching the target sound source from the audio generation model, wherein the audio generation model includes a plurality of sub-models, one sound source matches one sub-model, and the audio generation model is obtained by the model training method according to any one of claims 1 to 7; Target text data is acquired, and the target text data is input into the target sub-model to generate audio data simulating the target sound source.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory in communication with the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the model training method described in any one of claims 1-7, or can execute the audio generation method described in claim 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model training method described in any one of claims 1 to 7, or implement the audio generation method described in claim 8 when executed.

Images