Speech training method, device, equipment and storage medium based on deep learning

Abstract

The present invention discloses a speech training method, device, computer equipment and storage medium based on deep learning, which are applied to the field of artificial intelligence technology, and provide a method for training a speech synthesis model through a teacher-student neural network, which can train the speech synthesis model efficiently, quickly and with low resource consumption. The method provided by the present invention includes: encoding a first phoneme sequence to obtain a first phoneme encoding value; performing a time prediction process on the first phoneme encoding value to obtain a first pronunciation time prediction value; performing an expansion process on each phoneme in the first phoneme sequence to obtain an expansion feature of each phoneme in the first phoneme sequence; transforming the expansion feature of each phoneme in the first phoneme sequence into a first Mel spectrum value; training the student neural network through the hidden variables provided by the trained teacher neural network and the first Mel spectrum value, and training until the first loss function of the student neural network converges, to obtain a trained student neural network.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a voice training method, device, computer equipment and storage medium based on deep learning. Background Art

[0002] Most of the existing deep learning-based speech synthesis technologies are based on sequence-to-sequence (seq2seq) solutions, such as Tacotron2, which significantly improve the effect of speech synthesis compared to traditional statistical parameter model algorithms. However, relatively speaking, the training of sequence-to-sequence model systems requires a large amount of training data sets and computing resources to learn the model, and it is also difficult to achieve efficient speech synthesis in the inference stage. Some systems try to use different model structure techniques based on sequence-to-sequence models to reduce the pressure on computing resources. For example, by using convolutional neural networks in the encoding-decoding stage, this model can be trained quickly, but the problem is that sequence reasoning is still required, which is relatively inefficient. In order to avoid the serialized reasoning stage, some models use the self-attention mechanism to parallelize the structure of spectrum generation, but the training stage of the attention layer is very difficult and time-consuming. So far, there is a lack of a speech synthesis model that can achieve efficient training, efficient reasoning and high quality at the same time. Summary of the invention

[0003] The embodiments of the present invention provide a speech training method, apparatus, computer equipment and storage medium based on deep learning to solve the current technical problem of lacking a speech synthesis model that can simultaneously achieve efficient training, efficient reasoning and high quality.

[0004] In one aspect of the present invention, a speech training method based on deep learning is provided, comprising the following steps:

[0005] Encoding the first phoneme sequence to obtain a first phoneme encoding value;

[0006] Performing duration prediction processing on the first phoneme encoding value to obtain a first pronunciation duration prediction value;

[0007] Performing an expansion process on each phoneme in the first phoneme sequence based on the first pronunciation duration prediction value to obtain an expansion feature of each phoneme in the first phoneme sequence;

[0008] Transforming the extended feature of each phoneme in the first phoneme sequence into a first Mel-spectrogram value;

[0009] The student neural network is trained by using the hidden variables and the first Mel spectrum value provided by the trained teacher neural network. When the first loss function of the student neural network converges, a trained student neural network is obtained.

[0010] In another aspect of the present invention, a speech training device based on deep learning is provided, comprising the following modules:

[0011] A first phoneme encoding module, used for encoding a first phoneme sequence to obtain a first phoneme encoding value;

[0012] A duration prediction processing module, used for performing duration prediction processing on the first phoneme encoding value to obtain a first pronunciation duration prediction value;

[0013] An expansion processing module, used for performing expansion processing on each phoneme in the first phoneme sequence based on the first pronunciation duration prediction value to obtain an expansion feature of each phoneme in the first phoneme sequence;

[0014] A first Mel-spectrogram value conversion module, used for converting the extended feature of each phoneme in the first phoneme sequence into a first Mel-spectrogram value;

[0015] The student neural network training module is used to train the student neural network through the hidden variables and the first Mel spectrum value provided by the trained teacher neural network, and obtain the trained student neural network when the first loss function of the student neural network converges.

[0016] In another aspect of the present invention, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned deep learning-based speech training method when executing the computer program.

[0017] In another aspect of the present invention, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned deep learning-based speech training method are implemented.

[0018] The speech training method, device, computer equipment and storage medium based on deep learning provided by the present invention can solve the technical problem of the current lack of a speech synthesis model that can simultaneously achieve efficient training, efficient reasoning and high quality. Specifically, the sample data used for the training of the student neural network is simultaneously input into a pre-trained teacher neural network, and the teacher neural network provides hidden variables and reference Mel spectrum values ​​and supervises the machine learning process of the student neural network to improve the training and reasoning efficiency, and reduce the demand for hardware resources, while maintaining a good training effect as much as possible. Through the deep learning model of the teacher-student neural network provided by the present invention, since the teacher neural network has been pre-trained, the system resources occupied are not many, and the structure of the student neural network is relatively simple, and the system resources occupied during training are not many, so it can be trained on a single GPU resource, and the trained student neural network can synthesize speech in real time on the CPU due to its simple structure. The advantages of this system can be quickly applied in a variety of speech synthesis scenarios to provide an end-to-end speech synthesis solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0020] Figure 1 is a schematic diagram of an application environment of a speech training method based on deep learning in one embodiment of the present invention;

[0021] Figure 2 is a flow chart of a speech training method based on deep learning in one embodiment of the present invention;

[0022] Figure 3 is a flow chart of a training method of a teacher neural network in a speech training method based on deep learning in one embodiment of the present invention;

[0023] Figure 4 is a flow chart of a method for generating latent variables in a speech training method based on deep learning in one embodiment of the present invention;

[0024] Figure 5 It is a structural schematic diagram of a speech training apparatus based on deep learning in one embodiment of the present invention;

[0025] Figure 6 is a schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part rather than all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0027] In a specific embodiment, as shown in the figure, a voice training method based on deep learning is provided, which supervises and trains a student neural network through a trained teacher neural network. The above student network includes a phoneme encoder, a pronunciation duration predictor, and a decoder. Specifically, the training of the student neural network includes the following steps:

[0028] S101: Encode the first phoneme sequence to obtain a first phoneme encoding value.

[0029] A phoneme is the smallest speech unit divided according to the natural attributes of speech. Analyzing based on the pronunciation actions in a syllable, one action constitutes one phoneme. For example, a single word is decomposed into multiple syllables, and then each syllable is further decomposed into corresponding multiple phonemes. Further, for example, for the word sequence "peace", it has two corresponding syllables, namely "ping" and "an"; the syllable "ping" can be further decomposed into phonemes "p" and "ing", and the phoneme "an" can be further decomposed into phonemes "a" and "n". In the embodiments of the present application, in Chinese, one Chinese character corresponds to one syllable; in English, one word corresponds to one syllable, and the same applies to other languages.

[0030] The first phoneme sequence is the phoneme sequence used to train the student neural network, and it is necessary to encode it to obtain the first phoneme encoding value. Specifically, the first phoneme sequence is transformed and compressed into a vector with a fixed length. Specifically, the first phoneme sequence is transformed and compressed through the phoneme encoder in the student neural network. The above phoneme encoder has a four-layer structure: the first layer includes an Embedding Layer, a Fully Connected Layer, and a Rectified Linear Unit (ReLU); the second layer includes a one-dimensional Convolutional Neural Networks (CNN) layer; the third layer includes a rectified linear unit; the fourth layer includes Batch Normalization.

[0031] S102: Perform duration prediction processing on the first phoneme encoding value to obtain a first pronunciation duration prediction value.

[0032] According to the encoded first phoneme code value, the pronunciation duration predictor of the student neural network is used to predict the pronunciation duration of each phoneme in the first phoneme code value. Specifically, the pronunciation duration predictor has a three-layer structure: the first layer includes a one-dimensional convolutional neural network layer; the second layer includes a linear rectification layer; and the third layer includes a batch normalization layer.

[0033] S103: performing expansion processing on each phoneme in the first phoneme sequence based on the first pronunciation duration prediction value to obtain an expansion feature of each phoneme in the first phoneme sequence.

[0034] Perform acoustic feature prediction processing on the extended features of each phoneme to obtain the acoustic features of each phoneme, and synthesize the acoustic features of each phoneme into text acoustic features. This can be achieved in the following ways: determine the context features corresponding to the extended features of each phoneme; perform linear transformation on the context features corresponding to the extended features of each phoneme to obtain the acoustic features of each phoneme; and concatenate the acoustic features of each phoneme according to the order of each phoneme in the target text to obtain the acoustic features.

[0035] In a specific embodiment, each phoneme in the first phoneme sequence is expanded based on the acoustic features extracted from the true value of the Mel spectrum corresponding to the first phoneme sequence by the trained teacher neural network to obtain the expanded features of each phoneme in the first phoneme sequence.

[0036] The teacher neural network extracts the extended features of each phoneme in the real language from the input first phoneme encoding value and the true value of the Mel spectrum corresponding to the first phoneme sequence as the extended features of each phoneme in the first phoneme sequence.

[0037] S104: transforming the extended feature of each phoneme in the first phoneme sequence into a first Mel-spectrogram value.

[0038] Since the human ear's perception of sound is nonlinear, in order to simulate the sensitivity of the human ear to the actual frequency, a Mel filter function is often added to the linear spectrum to convert it into a nonlinear Mel spectrum.

[0039] The extended features of each phoneme in the first phoneme sequence are input to the decoder through the student neural network and transformed into a first Mel spectrum value. Specifically, the decoder of the student neural network has a three-layer structure: the first layer includes a one-dimensional convolutional neural network layer; the second layer includes a linear rectification layer; and the third layer includes a batch normalization layer. A linear layer is also included after the decoder.

[0040] S105: The student neural network is trained by using the hidden variables and the first Mel-spectrum value provided by the trained teacher neural network, and the trained student neural network is obtained when the first loss function of the student neural network converges.

[0041] The training method of the teacher-student neural network is a type of transfer learning in machine learning. Transfer learning is to transfer the achievable performance of a trained model to another model. The latter has a relatively simple structure compared to the former. For the teacher-student neural network, the teacher neural network is often a more complex network with better performance and versatility, but more system resources are needed to train the teacher neural network. Therefore, in order to save the system resources required for training, the trained teacher neural network can be used to provide a soft target to guide another student neural network with a simpler structure and less system resource consumption to learn, so that the student neural model with a simple structure and less parameter calculation can also obtain similar performance to the teacher network through training.

[0042] The first phoneme sequence used to train the student neural network and the true value of the Mel spectrum corresponding to the first phoneme sequence are also input into the pre-trained teacher neural network. The teacher neural network can output hidden variables and the Mel spectrum value generated by the teacher neural network according to the first phoneme sequence to the student neural network as a training effect for evaluating the student neural network. When the first loss function of the student neural network is trained to converge, a trained student neural network is obtained.

[0043] In a specific embodiment, the first loss function for evaluating the training effect of the student neural network is the mean absolute error between the Mel spectrum value and the Mel spectrum prediction value:

[0044]

[0045] where f i is the Mel spectrum value generated by the teacher neural network according to the first phoneme sequence, g i is the Mel-spectrogram value generated by the student neural network based on the first phoneme sequence;

[0046] Or, the first loss function is the Huber loss function:

[0047]

[0048] Among them, y is the Mel spectrum value generated by the teacher neural network according to the first phoneme sequence, f(x) is the Mel spectrum value generated by the student neural network according to the first phoneme sequence, and δ is a hyperparameter preset according to the expected training effect before the start of training.

[0049] In a specific embodiment, the deep learning-based speech training method further includes, after step S105:

[0050] S111: Connect the trained student neural network to a pre-trained vocoder;

[0051] S112: Convert the input phoneme sequence into corresponding Mel spectrum values ​​through the trained student neural network;

[0052] S113: Convert the Mel spectrum value into speech through a vocoder.

[0053] The above-mentioned vocoder is a neural network that can convert Mel spectrum values ​​into speech recognizable by human ears. The vocoder can select existing neural networks such as WaveNET, MelGAN, WaveGlow, etc., and after pre-training, receive the Mel spectrum values ​​output by the trained student neural network and convert them into speech. In a preferred embodiment, the lighter and faster MelGAN is preferred as the vocoder.

[0054] The speech training method, device, computer equipment and storage medium based on deep learning provided by the present invention simultaneously input the sample data used for the training of the student neural network into a pre-trained teacher neural network, and the teacher neural network provides hidden variables and reference Mel spectrum values ​​and supervises the machine learning process of the student neural network to improve the training and reasoning efficiency, and reduce the demand for hardware resources, while maintaining a good training effect as much as possible. Through the deep learning model of the teacher-student neural network provided by the present invention, since the teacher neural network has been pre-trained, it does not occupy many system resources. At the same time, the structure of the student neural network is relatively simple, and the system resources occupied during training are not many, so it can be trained on a single GPU resource, and the trained student neural network can perform real-time synthesis of speech on the CPU due to its simple structure. The advantages of this system can be quickly applied in a variety of speech synthesis scenarios to provide an end-to-end speech synthesis solution.

[0055] In another specific embodiment, a method for pre-training a teacher neural network for supervising training of a student neural network is provided, wherein the teacher neural network includes a phoneme encoder, a spectral encoder, an attention processing mechanism, and a decoder. Specifically, the steps of pre-training the teacher neural network include:

[0056] S201: Encode the second phoneme sequence to obtain a second phoneme encoding value.

[0057] The second phoneme sequence is a phoneme sequence used to train the teacher neural network, and needs to be encoded to obtain a second phoneme encoding value. Specifically, the second phoneme sequence is transformed and compressed by the phoneme encoder in the teacher neural network. The above-mentioned phoneme encoder has a four-layer structure: the first layer includes an embedding layer; the second layer includes a fully connected layer; the third layer includes a linear rectification function; the fourth layer includes an N-layer gated residual (Residual Gated) structure and a void residual convolution network.

[0058] S202: Encode the real Mel-spectrogram value corresponding to the second phoneme sequence after shifting it to the left by a preset value to obtain a second Mel-spectrogram encoding value.

[0059] The teacher neural network also has a spectral encoder that provides contextual encoding of the spectral frame, taking into account the previous spectral frames. The spectral encoder consists of fully connected layers, linear rectification functions, and an N-layer gated residual structure. First, the fully connected layers and linear rectification functions are applied to each frame of the input spectrum. This encoding result is input to the N-layer gated residual structure and a more fine-grained gated residual network. The true Mel-spectrogram values ​​corresponding to the second phoneme sequence input by the teacher neural network are shifted to the left by one position at the time of input, and the model can be used to predict the next spectral frame based on the current input phoneme and the previous spectral frame.

[0060] S203: Perform attention mechanism processing on the second phoneme code value and the second Mel-spectrogram code value to obtain the attention-added second phoneme code value and the second Mel-spectrogram code value.

[0061] The attention mechanism uses a dot product attention mechanism, in which the second phoneme encoding value includes the output of the phoneme encoder and the sum of the output of the phoneme encoder and the phoneme encoding. The second Mel spectrum encoding value is the output of the spectrum encoder. The attention score is the weighted mean of the output of the phoneme encoder and the sum of the phoneme encoding vector, and the weight is the matching value of the output of the phoneme encoder and the sum of the phoneme encoding and the output of the spectrum encoder. In this way, the model will tend to select phonemes related to the next spectrum frame.

[0062] S204: The attention-added second phoneme encoding value and the second Mel-spectrogram true encoding value are transformed into a second Mel-spectrogram value.

[0063] The input of the decoder of the teacher neural network is the output of the encoder and the sum of the attention scores, which is then passed through N layers of gated residual convolutional networks and convolutional layers with linear rectification functions to obtain the correct number of channels, and finally input into a sigmoid prediction layer to obtain the predicted spectral value.

[0064] S205: Self-training the teacher neural network according to the real Mel-spectrogram values ​​and the second Mel-spectrogram values ​​corresponding to the second phoneme sequence until the second loss function of the teacher neural network converges to obtain a trained teacher neural network.

[0065] The teacher network performs self-training using the second Mel-spectrogram values ​​generated according to the second phoneme sequence and the true Mel-spectrogram values ​​corresponding to the second phoneme sequence as training data until the second loss function converges.

[0066] In a specific embodiment, the second loss function for evaluating the self-training effect of the teacher neural network is the mean absolute error between the true Mel spectrum value and the second Mel spectrum value corresponding to the second phoneme sequence:

[0067]

[0068] where f i is the true Mel-spectrogram value corresponding to the second phoneme sequence, g i is the Mel-spectrogram value generated by the teacher neural network based on the second phoneme sequence.

[0069] In another specific embodiment, the step of generating latent variables from a pre-trained teacher neural network includes:

[0070] S301: The trained teacher neural network encodes the first phoneme sequence to obtain a third phoneme encoding value;

[0071] S302: The trained teacher neural network encodes the Mel spectrum true value corresponding to the first phoneme sequence after shifting it to the left by a preset value to obtain a third Mel spectrum encoding value;

[0072] S303: The trained teacher neural network performs attention mechanism processing on the third phoneme coding value and the third Mel spectrum true coding value to obtain the third phoneme coding value and the third Mel spectrum true coding value of the attention sum;

[0073] S304: transforming the third phoneme code value and the third Mel spectrum code value obtained by adding attention to the third phoneme code value into a third Mel spectrum value;

[0074] S305: The third phoneme coding value obtained by adding attention, the third Mel spectrum true coding value, and the third Mel spectrum value are output as latent variables to the student neural network through the trained teacher neural network.

[0075] The above steps S301-304 are similar to steps S201-204 of pre-training the teacher neural network. The difference is that the input is the first phoneme sequence used to train the student neural network and the true value of the Mel spectrum corresponding to the first phoneme sequence. The other steps are the same as the steps of pre-training the teacher neural network, which will not be repeated here.

[0076] The above step S305 is to convert the input first phoneme sequence and the true value of the Mel spectrum corresponding to the first phoneme sequence into the third phoneme coding value of the attention summation and the third Mel spectrum true coding value, and the third Mel spectrum value as a hidden variable to output to the student neural network. The above hidden variable is used to provide a soft target to a student neural network with a simpler structure to guide the student neural network to learn, so as to improve the learning speed and efficiency.

[0077] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0078] In another embodiment, a speech training device 100 based on deep learning is provided, which corresponds to the speech training method based on deep learning in the above embodiment. The device includes the following modules: a first phoneme encoding module 101, a duration prediction processing module 102, an expansion processing module 103, a first Mel spectrum value conversion module 104, and a student neural network training module 105.

[0079] A first phoneme encoding module 101, used to encode a first phoneme sequence to obtain a first phoneme encoding value;

[0080] The duration prediction processing module 102 is used to perform duration prediction processing on the first phoneme encoding value to obtain a first pronunciation duration prediction value;

[0081] An expansion processing module 103, configured to perform expansion processing on each phoneme in the first phoneme sequence based on the first pronunciation duration prediction value to obtain an expansion feature of each phoneme in the first phoneme sequence;

[0082] A first Mel-spectrogram value conversion module 104, configured to convert the extended feature of each phoneme in the first phoneme sequence into a first Mel-spectrogram value;

[0083] The student neural network training module 105 is used to train the student neural network using the latent variables and the first Mel spectrum value provided by the trained teacher neural network, and obtain the trained student neural network when the first loss function of the student neural network converges.

[0084] In another embodiment, the teacher neural network in the deep learning-based speech training device includes the following modules:

[0085] A second phoneme encoding module, used for encoding a second phoneme sequence to obtain a second phoneme encoding value;

[0086] A second Mel spectrum encoding module is used to encode the real Mel spectrum value corresponding to the second phoneme sequence after shifting it to the left by a preset value to obtain a second Mel spectrum encoding value;

[0087] An attention mechanism processing module, used for performing attention mechanism processing on the second phoneme encoding value and the second Mel spectrum encoding value to obtain the attention-added second phoneme encoding value and the second Mel spectrum encoding value;

[0088] A second Mel spectrum value conversion module, used for converting the attention-added second phoneme encoding value and the second Mel spectrum true encoding value into a second Mel spectrum value;

[0089] The teacher neural network self-training module is used to self-train the teacher neural network according to the real Mel spectrum and the second Mel spectrum corresponding to the second phoneme sequence until the second loss function of the teacher neural network converges to obtain a trained teacher neural network.

[0090] In another specific embodiment, the student neural network training module 105 in the deep learning-based speech training device further includes:

[0091] A third phoneme encoding unit is used to encode the first phoneme sequence through the trained teacher neural network to obtain a third phoneme encoding value;

[0092] A third Mel spectrum encoding unit is used for encoding the Mel spectrum true value corresponding to the first phoneme sequence after being shifted to the left by a preset value by the trained teacher neural network to obtain a third Mel spectrum encoding value;

[0093] An attention mechanism processing unit, wherein the trained teacher neural network performs attention mechanism processing on the third phoneme coding value and the third Mel spectrum true coding value to obtain the third phoneme coding value and the third Mel spectrum true coding value of the attention sum;

[0094] A third Mel-spectrogram value conversion unit, configured to convert the attention-added third phoneme encoding value and the third Mel-spectrogram encoding value into a third Mel-spectrogram value;

[0095] A latent variable output unit is used to output the third phoneme encoding value of the attention sum and the third Mel spectrum real encoding value, and the third Mel spectrum value as the latent variable to the student neural network through the trained teacher neural network.

[0096] In another specific embodiment, the first loss function selects the mean absolute error between the first Mel-spectrum value and the third Mel-spectrum value; or, the first loss function selects the Huber loss function.

[0097] In another specific embodiment, the expansion processing module 103 is specifically used to perform expansion processing on each phoneme in the first phoneme sequence according to the features extracted by the trained teacher neural network from the true value of the Mel spectrum corresponding to the first phoneme sequence, so as to obtain the expanded features of each phoneme in the first phoneme sequence.

[0098] In another specific embodiment, the deep learning-based speech training device 100 further includes:

[0099] A vocoder connection module, for connecting the trained student neural network to a pre-trained vocoder;

[0100] An input phoneme sequence conversion module is used to convert the input phoneme sequence into corresponding Mel spectrum values ​​through the trained student neural network;

[0101] A language output module is used to convert the Mel spectrum value into speech through the vocoder.

[0102] The meaning of "first" and "second" in the above modules / units is only to distinguish different modules / units, and is not used to define which module / unit has a higher priority or other limiting meanings. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of modules in this application is only a logical division, and there may be other division methods when implemented in actual applications.

[0103] For the specific definition of the voice training device based on deep learning, please refer to the definition of the voice training method based on deep learning above, which will not be repeated here. Each module in the above-mentioned voice training device based on deep learning can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0104] In one embodiment, a computer device is provided. The computer device may be a server or a workstation, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the speech training method based on deep learning. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a speech training method based on deep learning is implemented.

[0105] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the deep learning-based speech training method in the above embodiment are implemented, such as Figure 2 Alternatively, when the processor executes the computer program, the functions of each module / unit of the speech training device based on deep learning in the above embodiment are realized, for example, Figure 5 The functions of modules 101 to 105 are shown in Figure 1. To avoid repetition, they will not be described here.

[0106] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and various interfaces and lines are used to connect various parts of the entire computer device.

[0107] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the computer device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, video data, etc.), etc.

[0108] The memory may be integrated into the processor or may be arranged separately from the processor.

[0109] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the speech training method based on deep learning in the above embodiment are implemented, such as Figure 2 Alternatively, when the computer program is executed by a processor, the functions of each module / unit of the speech training device based on deep learning in the above-mentioned embodiment are realized, for example, Figure 5 The functions of modules 101 to 105 are shown in Figure 1. To avoid repetition, they will not be described here.

[0110] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0111] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0112] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A speech training method based on deep learning, characterized in that: The following steps are involved: Encoding the first phoneme sequence to obtain a first phoneme encoding value; Performing duration prediction processing on the first phoneme encoding value to obtain a first pronunciation duration prediction value; Performing an expansion process on each phoneme in the first phoneme sequence based on the first pronunciation duration prediction value to obtain an expansion feature of each phoneme in the first phoneme sequence; transforming the extended feature of each phoneme in the first phoneme sequence into a first Mel-spectrogram value; The student neural network is trained by using the hidden variables provided by the trained teacher neural network and the first Mel spectrum value, and the trained student neural network is obtained when the first loss function of the student neural network converges; In the step of training the student neural network by using the latent variables provided by the trained teacher neural network and the first Mel spectrum value, and training until the first loss function of the student neural network converges, the trained student neural network is obtained, which further includes: encoding the first phoneme sequence by the trained teacher neural network to obtain a third phoneme encoding value; encoding the Mel spectrum true value corresponding to the first phoneme sequence after shifting it to the left by a preset value to obtain a third Mel spectrum encoding value; performing attention mechanism processing on the third phoneme encoding value and the third Mel spectrum true encoding value by the trained teacher neural network to obtain an attention-added third phoneme encoding value and a third Mel spectrum true encoding value; transforming the attention-added third phoneme encoding value and the third Mel spectrum encoding value into a third Mel spectrum value; and outputting the attention-added third phoneme encoding value and the third Mel spectrum true encoding value and the third Mel spectrum value as the latent variables to the student neural network through the trained teacher neural network.

2. The speech training method based on deep learning according to claim 1, characterized in that: The steps of training the teacher neural network include: Encoding the second phoneme sequence to obtain a second phoneme encoding value; The real Mel-spectrogram value corresponding to the second phoneme sequence is shifted to the left by a preset value and then encoded to obtain a second Mel-spectrogram encoding value; Performing attention mechanism processing on the second phoneme encoding value and the second Mel-spectrogram encoding value to obtain the second phoneme encoding value and the second Mel-spectrogram encoding value with attention added; Transforming the attention-added second phoneme encoding value and the second Mel-spectrogram true encoding value into a second Mel-spectrogram value; The teacher neural network is self-trained according to the real Mel spectrum value corresponding to the second phoneme sequence and the second Mel spectrum value until the second loss function of the teacher neural network converges to obtain the trained teacher neural network.

3. The speech training method based on deep learning according to claim 1, characterized in that: The first loss function selects the mean absolute error between the first Mel spectrum value and the third Mel spectrum value; Alternatively, the first loss function is a Huber loss function.

4. The speech training method based on deep learning according to claim 1, characterized in that: The step of performing expansion processing on each phoneme in the first phoneme sequence based on the first pronunciation duration prediction value to obtain an expansion feature of each phoneme in the first phoneme sequence specifically includes: Each phoneme in the first phoneme sequence is expanded according to the features extracted from the true value of the Mel spectrum corresponding to the first phoneme sequence by the trained teacher neural network to obtain the expanded features of each phoneme in the first phoneme sequence.

5. The deep learning-based speech training method according to claim 1, characterized in that: The step of training the student neural network by using the hidden variables provided by the trained teacher neural network and the first Mel spectrum value until the first loss function of the student neural network converges to obtain the trained student neural network also includes: connecting the trained student neural network to a pre-trained vocoder; The input phoneme sequence is converted into corresponding Mel spectrum values ​​through the trained student neural network; The mel-spectrogram value is converted into speech by the vocoder.

6. A speech training device based on deep learning, characterized in that: Includes the following modules: A first phoneme encoding module, used for encoding a first phoneme sequence to obtain a first phoneme encoding value; A duration prediction processing module, used for performing duration prediction processing on the first phoneme encoding value to obtain a first pronunciation duration prediction value; An expansion processing module, configured to perform expansion processing on each phoneme in the first phoneme sequence based on the first pronunciation duration prediction value to obtain an expansion feature of each phoneme in the first phoneme sequence; A first Mel-spectrogram value conversion module, used for converting the extended feature of each phoneme in the first phoneme sequence into a first Mel-spectrogram value; A student neural network training module, used for training the student neural network by using the latent variables provided by the trained teacher neural network and the first Mel spectrum value, and obtaining the trained student neural network when the first loss function of the student neural network converges; The student neural network training module also includes a third phoneme encoding unit, a third Mel spectrum encoding unit, an attention mechanism processing unit, a third Mel spectrum value conversion unit and a hidden variable output unit; The third phoneme encoding unit is used for encoding the first phoneme sequence by the trained teacher neural network to obtain a third phoneme encoding value; The third Mel spectrum encoding unit is used for the trained teacher neural network to encode the Mel spectrum true value corresponding to the first phoneme sequence after shifting it to the left by a preset value to obtain a third Mel spectrum encoding value; The attention mechanism processing unit is used for the trained teacher neural network to perform attention mechanism processing on the third phoneme coding value and the third Mel spectrum true coding value to obtain the attention-added third phoneme coding value and the third Mel spectrum true coding value; The third Mel-spectrogram value conversion unit is used to convert the attention-added third phoneme encoding value and the third Mel-spectrogram encoding value into a third Mel-spectrogram value; The latent variable output unit is used to output the third phoneme encoding value of the attention sum and the third Mel spectrum true encoding value, and the third Mel spectrum value as the latent variable to the student neural network through the trained teacher neural network.

7. The deep learning-based speech training device according to claim 6, characterized in that: The teacher neural network includes the following modules: A second phoneme encoding module, used for encoding a second phoneme sequence to obtain a second phoneme encoding value; A second Mel-spectrogram encoding module, used for encoding the real Mel-spectrogram value corresponding to the second phoneme sequence after shifting it to the left by a preset value to obtain a second Mel-spectrogram encoding value; an attention mechanism processing module, configured to perform attention mechanism processing on the second phoneme encoding value and the second Mel-spectrogram encoding value to obtain the second phoneme encoding value and the second Mel-spectrogram encoding value with attention added; A second Mel-spectrogram value conversion module, used for converting the attention-added second phoneme encoding value and the second Mel-spectrogram true encoding value into a second Mel-spectrogram value; The teacher neural network self-training module is used to self-train the teacher neural network according to the real Mel spectrum corresponding to the second phoneme sequence and the second Mel spectrum until the second loss function of the teacher neural network converges to obtain the trained teacher neural network.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the deep learning-based speech training method as described in any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the deep learning-based speech training method as described in any one of claims 1 to 5 are implemented.

Images