Vocoder training method, device, equipment and storage medium

Abstract

This disclosure provides a vocoder training method, apparatus, device, and storage medium, relating to the field of computer technology, specifically to artificial intelligence fields such as speech synthesis and deep learning. The vocoder training method includes obtaining existing data, including existing audio; performing data augmentation processing on the existing audio to obtain augmented audio; and training the vocoder based on the existing audio and the augmented audio. This disclosure can improve the performance of the vocoder.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically to artificial intelligence fields such as speech synthesis and deep learning, and more particularly to a training method, apparatus, device, and storage medium for a vocoder. Background Art

[0002] Text to Speech (TTS) converts text into audio. This process typically uses a speech synthesis model to process text to generate the corresponding audio.

[0003] The speech synthesis model may include an acoustic model and a vocoder. The acoustic model converts the input speech into acoustic features, and the vocoder converts the input acoustic features into audio.

[0004] Due to the limited training data corresponding to the vocoder, the effect of the vocoder is not ideal. Summary of the Invention

[0005] The present disclosure provides a vocoder training method, apparatus, device, and storage medium.

[0006] According to one aspect of the present disclosure, a method for training a vocoder is provided, comprising: obtaining existing data, the existing data comprising existing audio; performing data enhancement processing on the existing audio to obtain expanded audio; and training a vocoder based on the existing audio and the expanded audio.

[0007] According to another aspect of the present disclosure, a vocoder training device is provided, comprising: an acquisition module for acquiring existing data, wherein the existing data includes existing audio; a processing module for performing data enhancement processing on the existing audio to obtain augmented audio; and a training module for training a vocoder based on the existing audio and the augmented audio.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method as described in any of the above aspects.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods according to any one of the above aspects.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of the above aspects.

[0011] According to the technical solution disclosed in the present invention, the effect of the vocoder can be improved.

[0012] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0014] Figure 1 is a schematic diagram according to a first embodiment of the present disclosure;

[0015] Figure 2 is a schematic diagram according to a second embodiment of the present disclosure;

[0016] Figure 3 is a schematic diagram according to a third embodiment of the present disclosure;

[0017] Figure 4 is a schematic diagram according to a fourth embodiment of the present disclosure;

[0018] Figure 5 is a schematic diagram according to a fifth embodiment of the present disclosure;

[0019] Figure 6 is a schematic diagram according to a sixth embodiment of the present disclosure;

[0020] Figure 7 2 is a schematic diagram of an electronic device for implementing the vocoder training method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] Generally speaking, to improve model performance, a large amount of training data is usually used. However, for vocoders, obtaining large amounts of training data is difficult, and vocoder performance is poor based on limited training data.

[0023] In order to improve the effect of the vocoder, the present disclosure provides the following embodiments.

[0024] Figure 1 FIG1 is a schematic diagram according to a first embodiment of the present disclosure. This embodiment provides a method for training a vocoder, the method comprising:

[0025] 101. Acquire existing data, where the existing data includes existing audio.

[0026] 102. Perform data enhancement processing on the existing audio to obtain expanded audio.

[0027] 103. Train a vocoder based on the existing audio and the expanded audio.

[0028] Among them, the speech synthesis method can be applied to different speech synthesis scenarios, such as voice navigation, call center, novel voice playback, smart speaker voice interaction, etc.

[0029] The existing audio may be obtained from an existing training set. For example, corresponding to different scenarios, such as a voice navigation scenario, the existing audio may be obtained from an existing training set of the voice navigation scenario.

[0030] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0031] Because existing data is limited, for example, the amount of existing audio data for voice navigation scenarios is relatively small. To increase the data volume, data augmentation can be performed on the existing audio to generate augmented audio. Both the existing audio and the augmented audio can then be used as audio samples for vocoder training.

[0032] like Figure 2 As shown, the speech synthesis model may include an acoustic model 201 and a vocoder 202. The acoustic model is used to process the input text to obtain acoustic features corresponding to the text, such as Mel spectrum; the vocoder is used to process the input acoustic features to obtain audio corresponding to the text.

[0033] The acoustic model 201 and the vocoder 202 may both be deep neural network models.

[0034] This embodiment describes the training process of a vocoder.

[0035] See also Figure 2The input of the vocoder 202 is acoustic features and the output is audio. Therefore, when training the vocoder, the acoustic feature samples and audio samples can be obtained first, and then the acoustic feature samples and audio samples can be used to train the vocoder.

[0036] The audio samples may include the above-mentioned existing audio and expanded audio.

[0037] Since the augmented audio is obtained by performing data augmentation processing on the existing audio, the augmented audio can be obtained based on the existing audio. The training data during vocoder training includes not only the existing audio but also the augmented audio. Therefore, the training data is enriched and the amount of training data is increased, thereby improving the vocoder effect based on a small amount of existing audio.

[0038] In some embodiments, the data enhancement processing of the existing audio to obtain the expanded audio includes at least one of the following items: adjusting the speed of the existing audio to obtain speed-adjusted audio, and using the speed-adjusted audio as the expanded audio; adjusting the fundamental frequency of the existing audio to obtain fundamental frequency-adjusted audio, and using the fundamental frequency-adjusted audio as the expanded audio; adjusting the volume of the existing audio to obtain volume-adjusted audio samples, and using the volume-adjusted audio samples as the expanded audio.

[0039] That is, the expanded audio can be obtained by adjusting one or more of the speed, fundamental frequency, and volume of the existing audio.

[0040] Assume that the existing audio is represented by x1, x2, x3..., where xi (i=1, 2, 3...) (i is a time index, which is a positive integer) is an audio unit.

[0041] Speed ​​adjustment can be achieved by dropping or inserting some audio units. Speed ​​adjustment can include audio acceleration or deceleration. Acceleration can be achieved by dropping some audio units, while deceleration can be achieved by inserting some audio units. Dropping or inserting some audio units can be achieved through filtering.

[0042] For fundamental frequency adjustment, a synchronous overlap-add method can be used to convert the existing audio into the frequency domain first, and then perform an overall left or right shift on the amplitude in the frequency domain to achieve down-tuning or up-tuning of the fundamental frequency of the audio.

[0043] For volume adjustment, the volume can be increased or decreased by changing the scaling factor of the audio amplitude.

[0044] Specifically, the speed, base frequency, and volume can be adjusted using SoX (Sound eXchange) software, which is software that can adjust audio formats.

[0045] For example, see Figure 3 , using SoX software to process the input existing audio and output expanded audio.

[0046] Using SoX software, you can adjust one or more of the speed, audio frequency, and volume of existing audio.

[0047] For example, for speed, you can augment the data by changing the speed of existing audio by 0.8, 0.9, 1.0, 1.1, or 1.2 times. For fundamental frequency, you can augment the data by changing the fundamental frequency by 0.8, 0.9, 1.0, 1.1, or 1.2 times. For volume, you can augment the data by changing the volume by 0.8, 0.9, 1.0, 1.1, or 1.2 times.

[0048] By adjusting one or more of the speed, fundamental frequency, and volume of existing audio, expanded audio can be obtained, thereby increasing the amount of training data.

[0049] After obtaining the augmented audio, the vocoder can be trained based on the existing audio and the augmented audio.

[0050] The vocoder is a deep neural network model. Generally speaking, the training process of a deep neural network model includes multiple update processes to generate a final model from an initial model.

[0051] That is, for the vocoder, the existing data also includes acoustic feature samples, the existing audio and the expanded audio are used as audio samples, and the training of the vocoder based on the existing audio and the expanded audio includes: performing at least one step of model parameter updating process based on the acoustic feature samples and the audio samples to generate the vocoder.

[0052] The acoustic feature samples may be obtained based on existing data. For example, after an existing text sample is processed by an acoustic model, the acoustic features output by the acoustic model are used as acoustic feature samples. Alternatively, the acoustic feature samples may be directly obtained from the existing data.

[0053] When training a vocoder, you can start from an initial vocoder, perform at least one step of model parameter update process, obtain final model parameters, and use the vocoder with the final model parameters as the final vocoder, that is, the vocoder generated after training.

[0054] The initial vocoder may be a pre-trained model, and the pre-trained model may be a pre-trained model that has been trained using a large amount of data in various fields.

[0055] This embodiment can use a small amount of existing data in a specific field, such as a small amount of existing data in the field of voice navigation, on the basis of the pre-trained model to fine-tune the existing pre-trained model to obtain the final vocoder.

[0056] Here, at least one step may also be referred to as at least one round, at least one time, etc. The total number of steps in the update process may be set, for example, 100,000 times.

[0057] The updating process of each step is similar, which is to update the existing model parameters, which can also be called weights.

[0058] During the update process, each step can be treated as the current step, allowing the model parameters for each step to be updated sequentially. The existing model parameters for the current step can be referred to as the initial model parameters, or the unupdated model parameters. Subsequently, the initial model parameters are updated to obtain updated model parameters. The updated model parameters for the current step can be used as the initial model parameters for the next step after the current step. In other words, the initial model parameters for the current step are the updated model parameters from the previous step.

[0059] The vocoder can be a deep neural network model including at least one sub-model. For example, the vocoder is WaveRNN. WaveRNN is an industrial-grade mainstream neural network framework vocoder and a multi-subband small sparse model.

[0060] In some embodiments, the existing data also includes acoustic feature samples, the existing audio and the augmented audio are used as audio samples, the vocoder includes at least one sub-model, the training includes at least one model parameter updating process, and the training of the vocoder based on the existing audio and the augmented audio includes: performing at least one model parameter updating process based on the acoustic feature samples and the audio samples to generate the vocoder; wherein, for the at least one model parameter updating process, performing: sub-band splitting of the audio sample to obtain at least one real sub-band audio; using the initial model parameters of the at least one sub-model, processing the acoustic feature sample to output at least one predicted sub-band audio; constructing a loss function based on the at least one real sub-band audio and the at least one predicted sub-band audio; adjusting the initial model parameters of the at least one sub-model based on the loss function to obtain the adjusted model parameters of the at least one sub-model; and determining the updated model parameters of the at least one sub-model based on the adjusted model parameters of the at least one sub-model.

[0061] Among them, combined Figure 2The input of the vocoder is acoustic features and the output is audio. Therefore, when training the vocoder, acoustic feature samples and audio samples can be used as training data. In this embodiment, the acoustic feature samples can be obtained from existing data, and the audio samples can include existing audio in the existing data, as well as extended audio obtained based on the existing audio.

[0062] For example, a given audio frequency can be divided into a preset number of sub-bands. For example, for a 16kHz audio frequency, which has 16,000 points per second, if it is divided into 8 sub-bands, the 16,000 points will be evenly divided into 2,000 points per sub-band. The 2,000-point signal corresponding to each sub-band is called a sub-band audio frequency.

[0063] like Figure 4 As shown in the figure, taking eight sub-band audio as an example, there are eight corresponding sub-models, represented by the first to eighth sub-models. Acoustic feature samples are input into each of the eight sub-models, and each sub-model outputs a predicted sub-band audio. Thus, eight predicted sub-band audios are obtained, represented by the first to eighth predicted sub-band audios.

[0064] In addition, after the audio sample is split, eight real sub-band audios can be obtained, which are represented by the first real sub-band audio to the eighth real sub-band audio. Among them, a pseudo quadrature mirror filter (PQMF) can be used to perform sub-band splitting on the audio sample.

[0065] Corresponding to the same sub-band, each predicted sub-band audio and each real sub-band audio can be constructed ( Figure 4 The corresponding loss functions (represented by circles in the figure) are represented by L1, L2, ..., L8, respectively. L1 and other loss functions can be cross-entropy loss functions or Gaussian loss functions. The average of these eight loss functions can then be calculated as the final loss function, i.e., L = (L1 + L2 + ... + L8) / 8.

[0066] After obtaining the final loss function L, the model parameters of each sub-model can be adjusted based on the loss function L, wherein the back propagation (BP) algorithm can be used to adjust the model parameters.

[0067] After the adjusted model parameters are obtained, updated model parameters may be obtained based on the adjusted model parameters.

[0068] By splitting the audio samples into sub-bands to obtain real sub-band audio, and using sub-models to obtain predicted sub-band audio, the model parameters of the sub-model can be adjusted based on the real sub-band audio and the predicted sub-band audio to obtain the final vocoder, which is suitable for the model parameter adjustment process of multi-sub-band vocoders.

[0069] In some embodiments, the at least one step includes a current step, and the determining of the updated model parameters of the at least one sub-model based on the adjusted model parameters of the at least one sub-model includes: if the number of steps of the current step is less than or equal to a first preset value, using the adjusted model parameters of the at least one sub-model as the updated model parameters of the at least one sub-model; if the number of steps of the current step is greater than the first preset value, using the sparsification rule corresponding to the at least one sub-model to perform sparsification processing on the adjusted model parameters of the at least one sub-model to obtain the updated model parameters of the at least one sub-model.

[0070] For example, see Figure 5 For the current step, taking the model parameters W1 of the first sub-model as an example, for simplicity, assume that there are three model parameters in W1. The adjusted model parameters are represented by [a1, a2, a3]. If the number of steps in the current step (t) is less than or equal to the first preset value, the adjusted model parameters are used as the updated model parameters, that is, the updated model parameters are [a1, a2, a3]. Otherwise, the adjusted model parameters are sparsely processed and used as the updated model parameters. For more information about the sparse processing, please refer to the subsequent description.

[0071] The first preset value is, for example, 8000. Therefore, if the number of steps in the current step is 0 to 8000, the adjusted model parameters can be directly used as updated model parameters.

[0072] If the number of steps in the current step is greater than 8000, the adjusted model parameters can be sparsely processed to obtain updated model parameters.

[0073] The sparsification process refers to reducing the absolute value of the parameter value of the model parameter, or even setting the parameter value to 0.

[0074] By sparsifying the model parameters, the model training speed can be accelerated.

[0075] In some embodiments, the sparsification rule corresponding to the at least one sub-model is used to perform sparsification processing on the adjusted model parameters of the at least one sub-model to obtain the updated model parameters of the at least one sub-model, including: using the sparsification rule corresponding to the at least one sub-model to determine the model parameters to be processed among the adjusted model parameters; multiplying the parameter values ​​of the model parameters to be processed by preset coefficients, and keeping the parameter values ​​of the unprocessed model parameters other than the model parameters to be processed unchanged to obtain the updated model parameters of the at least one sub-model.

[0076] Among them, taking 8 sub-models as an example, the corresponding model parameters are represented by W1, W2, ..., W8. Generally speaking, these 8 groups of model parameters are sparse together, that is, these 8 groups of model parameters correspond to one sparsification rule, rather than each sub-model corresponding to a sparsification rule.

[0077] For example, if the sparsification rule is to sparsify 80% of the model parameters, then in the related art, 80% of the model parameters can be selected from 8 groups of model parameters for sparsification.

[0078] In this embodiment, each sub-model may correspond to a sparsification rule. Therefore, for each group of model parameters, 80% of the model parameters in the group of model parameters may be selected for sparsification.

[0079] Among them, 80% of the parameter values ​​with the smallest absolute values ​​can be selected as the model parameters to be processed.

[0080] See also Figure 5 Taking W1 as an example, assuming that a1 is the model parameter to be processed, the updated model parameter is [a1*k,a2,a3], where * represents the multiplication operation and k is the preset coefficient.

[0081] For the model parameter to be processed, the product of its parameter value and the preset coefficient can be used as the parameter value after the sparsification process. For example, if the preset coefficient is 0, then for a certain model parameter to be processed, after the sparsification process, its parameter value becomes 0.

[0082] By adopting the sparsification rules corresponding to the sub-models for sparsification processing, the model parameters of each sub-model can be processed separately instead of together, thereby achieving the independence of sparsification and improving the sparsification effect.

[0083] In some embodiments, multiplying the parameter value of the model parameter to be processed by a preset coefficient includes: if the number of steps of the current step is greater than the first preset value and less than or equal to the second preset value, multiplying the parameter value of the model parameter to be processed by a first preset coefficient, and the first preset coefficient is a value greater than 0 and less than 1; or, if the number of steps of the current step is greater than the second preset value, multiplying the parameter value of the model parameter to be processed by a second preset coefficient, and the second preset coefficient is 0; wherein the second preset value is greater than the first preset value.

[0084] For example, see Figure 5 If the number of steps (t) of the current step is greater than the first preset value and less than or equal to the second preset value, k is a value greater than 0 and less than 1. If t is greater than the second preset value, k = 0, that is, the parameter value of the model parameter to be processed is set to 0.

[0085] Among them, the first preset value can be 8000, and the second preset value can be 80000. When the current step is between 8000 and 80000, the parameter value of the model parameter is not directly set to 0, but multiplied by a coefficient based on the original parameter value; when the current step is greater than 80000, the parameter value of the model parameter can be directly set to 0.

[0086] Therefore, assuming that the first preset value is 8000 and the second preset value is 80000, and assuming that the model parameters adjusted based on the loss function are represented by a and the updated model parameters are represented by a', the two have the following relationship:

[0087]

[0088] Among them, 0<k<1 is the preset coefficient.

[0089] In the early stages of training, the model parameters (weights) will be relatively large. If the weights are directly reset to 0, the training will be more volatile.

[0090] By introducing the non-zero k mentioned above, weight fallback can be achieved, thereby achieving sparse smoothness, avoiding the oscillation problem in the early stage of training, and improving the stability of training. In addition, the learned information can be used in the model training process to improve the accuracy of the model.

[0091] Figure 6 is a schematic diagram according to the sixth embodiment of the present disclosure, which provides a training device for a vocoder. Figure 6 As shown, the device 600 includes: an acquisition module 601, a processing module 602 and a training module 603.

[0092] The acquisition module 601 is used to acquire existing data, which includes: existing audio; the processing module 602 is used to perform data enhancement processing on the existing audio to obtain expanded audio; the training module 603 is used to train the vocoder based on the existing audio and the expanded audio

[0093] In some embodiments, the processing module 602 is further used to perform one or more of the following items: adjusting the speed of the existing audio to obtain speed-adjusted audio, and using the speed-adjusted audio as the expanded audio; adjusting the fundamental frequency of the existing audio to obtain fundamental frequency-adjusted audio, and using the fundamental frequency-adjusted audio as the expanded audio; adjusting the volume of the existing audio to obtain volume-adjusted audio samples, and using the volume-adjusted audio samples as the expanded audio.

[0094] In some embodiments, the existing data also includes acoustic feature samples, the existing audio and the expanded audio are used as audio samples, the vocoder includes at least one sub-model, and the training module 603 is further used to: based on the acoustic feature samples and the audio samples, perform at least one step of model parameter updating process to generate the vocoder; wherein, for the at least one step of model parameter updating process, perform: sub-band splitting of the audio sample to obtain at least one real sub-band audio; using the initial model parameters of the at least one sub-model, processing the acoustic feature sample to output at least one predicted sub-band audio; constructing a loss function based on the at least one real sub-band audio and the at least one predicted sub-band audio; based on the loss function, adjusting the initial model parameters of the at least one sub-model to obtain the adjusted model parameters of the at least one sub-model; and determining the updated model parameters of the at least one sub-model based on the adjusted model parameters of the at least one sub-model.

[0095] In some embodiments, the at least one step includes the current step, and the training module 603 is further used to: if the number of steps of the current step is less than or equal to a first preset value, use the adjusted model parameters of the at least one sub-model as the updated model parameters of the at least one sub-model; if the number of steps of the current step is greater than the first preset value, use the sparsification rule corresponding to the at least one sub-model to perform sparsification processing on the adjusted model parameters of the at least one sub-model to obtain the updated model parameters of the at least one sub-model.

[0096] In some embodiments, the training module 603 is further used to: adopt the sparsification rule corresponding to the at least one sub-model to determine the model parameters to be processed in the adjusted model parameters; multiply the parameter values ​​of the model parameters to be processed by preset coefficients, and keep the parameter values ​​of the unprocessed model parameters other than the model parameters to be processed unchanged to obtain the updated model parameters of the at least one sub-model.

[0097] In some embodiments, the training module 603 is further used to: if the number of steps of the current step is greater than the first preset value and less than or equal to the second preset value, multiply the parameter value of the model parameter to be processed by a first preset coefficient, and the first preset coefficient is a value greater than 0 and less than 1; or, if the number of steps of the current step is greater than the second preset value, multiply the parameter value of the model parameter to be processed by a second preset coefficient, and the second preset coefficient is 0; wherein the second preset value is greater than the first preset value.

[0098] Since the augmented audio is obtained by performing data augmentation processing on the existing audio, the augmented audio can be obtained based on the existing audio. The training data during vocoder training includes not only the existing audio but also the augmented audio. Therefore, the training data is enriched and the amount of training data is increased, thereby improving the vocoder effect based on a small amount of existing audio.

[0099] By adjusting one or more of the speed, fundamental frequency, and volume of existing audio, expanded audio can be obtained, thereby increasing the amount of training data.

[0100] By splitting the audio samples into sub-bands to obtain real sub-band audio, and using sub-models to obtain predicted sub-band audio, the model parameters of the sub-model can be adjusted based on the real sub-band audio and the predicted sub-band audio to obtain the final vocoder, which is suitable for the model parameter adjustment process of multi-sub-band vocoders.

[0101] By sparsifying the model parameters, the model training speed can be accelerated.

[0102] By adopting the sparsification rules corresponding to the sub-models for sparsification processing, the model parameters of each sub-model can be processed separately instead of together, thereby achieving the independence of sparsification and improving the sparsification effect.

[0103] By introducing the non-zero k mentioned above, weight fallback can be achieved, thereby achieving sparse smoothness, avoiding the oscillation problem in the early stage of training, and improving the stability of training. In addition, the learned information can be used in the model training process to improve the accuracy of the model.

[0104] It can be understood that in the embodiments of the present disclosure, the same or similar contents in different embodiments can be referenced to each other.

[0105] It can be understood that the terms “first”, “second”, etc. in the embodiments of the present disclosure are only used for distinction and do not indicate the degree of importance, time sequence, etc.

[0106] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0107] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0108] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0109] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. Various programs and data required for the operation of the electronic device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0110] Multiple components in the electronic device 700 are connected to the I / O interface 705, including an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0111] The computing unit 701 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as the vocoder training method. For example, in some embodiments, the vocoder training method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the vocoder training method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the vocoder training method by any other appropriate means (e.g., by means of firmware).

[0112] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0114] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0116] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0117] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, establishing a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within the cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0118] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0119] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A vocoder training method comprising: Acquire existing data, the existing data including: existing audio; Performing data enhancement processing on the existing audio to obtain expanded audio; Training a vocoder based on the existing audio and the augmented audio; The existing data further includes acoustic feature samples, the existing audio and the augmented audio are used as audio samples, the vocoder includes at least one sub-model, and the training of the vocoder based on the existing audio and the augmented audio includes: Based on the acoustic feature samples and the audio samples, performing at least one step of a model parameter updating process to generate the vocoder; Wherein, for the at least one step of the model parameter updating process, execute: Performing sub-band splitting on the audio sample to obtain at least one real sub-band audio; Processing the acoustic feature samples using initial model parameters of the at least one sub-model to output at least one predicted sub-band audio; constructing a loss function based on the at least one real sub-band audio and the at least one predicted sub-band audio; Adjusting initial model parameters of the at least one sub-model based on the loss function to obtain adjusted model parameters of the at least one sub-model; determining updated model parameters of the at least one sub-model based on the adjusted model parameters of the at least one sub-model; The determining, based on the adjusted model parameters of the at least one sub-model, updated model parameters of the at least one sub-model comprises: If the number of steps of the current step in the at least one step is greater than a first preset value, performing a sparsification process on the adjusted model parameters of the at least one sub-model using a sparsification rule corresponding to the at least one sub-model to obtain updated model parameters of the at least one sub-model; The sub-models correspond to the sparsification rules one by one; The sparsification processing includes: when the number of steps in the current step is greater than the first preset value and less than or equal to the second preset value, multiplying the parameter value of the model parameter to be processed in the adjusted model parameters by a first preset coefficient, the first preset coefficient is a value greater than 0 and less than 1, and the second preset value is greater than the first preset value.

2. The method according to claim 1, wherein The performing data enhancement processing on the existing audio to obtain the expanded audio includes at least one of the following items: Adjusting the speed of the existing audio to obtain speed-adjusted audio, and using the speed-adjusted audio as the expanded audio; performing fundamental frequency adjustment on the existing audio to obtain audio after fundamental frequency adjustment, and using the audio after fundamental frequency adjustment as the expanded audio; The volume of the existing audio is adjusted to obtain a volume-adjusted audio sample, and the volume-adjusted audio sample is used as the expanded audio.

3. The method according to claim 1, wherein The determining, based on the adjusted model parameters of the at least one sub-model, updated model parameters of the at least one sub-model further includes: If the number of steps of the current step is less than or equal to a first preset value, the adjusted model parameters of the at least one sub-model are used as the updated model parameters of the at least one sub-model.

4. The method according to claim 1, wherein The adopting the sparsification rule corresponding to the at least one sub-model to perform sparsification processing on the adjusted model parameters of the at least one sub-model to obtain the updated model parameters of the at least one sub-model includes: Determine the model parameters to be processed in the adjusted model parameters by using the sparsification rule corresponding to the at least one sub-model; The parameter values ​​of the to-be-processed model parameters are multiplied by preset coefficients, and the parameter values ​​of the unprocessed model parameters other than the to-be-processed model parameters are kept unchanged to obtain updated model parameters of the at least one sub-model.

5. The method according to claim 4, wherein The multiplying the parameter value of the to-be-processed model parameter by a preset coefficient further includes: If the number of steps of the current step is greater than the second preset value, the parameter value of the model parameter to be processed is multiplied by a second preset coefficient, and the second preset coefficient is 0.

6. A vocoder training device comprising: An acquisition module, configured to acquire existing data, wherein the existing data includes: existing audio; A processing module, configured to perform data enhancement processing on the existing audio to obtain expanded audio; A training module, configured to train a vocoder based on the existing audio and the expanded audio; The existing data further includes acoustic feature samples, the existing audio and the expanded audio serve as audio samples, the vocoder includes at least one sub-model, and the training module is further configured to: Based on the acoustic feature samples and the audio samples, performing at least one step of a model parameter updating process to generate the vocoder; Wherein, for the at least one step of the model parameter updating process, execute: Performing sub-band splitting on the audio sample to obtain at least one real sub-band audio; Processing the acoustic feature samples using initial model parameters of the at least one sub-model to output at least one predicted sub-band audio; constructing a loss function based on the at least one real sub-band audio and the at least one predicted sub-band audio; Adjusting initial model parameters of the at least one sub-model based on the loss function to obtain adjusted model parameters of the at least one sub-model; determining updated model parameters of the at least one sub-model based on the adjusted model parameters of the at least one sub-model; The determining, based on the adjusted model parameters of the at least one sub-model, updated model parameters of the at least one sub-model comprises: If the number of steps of the current step in the at least one step is greater than a first preset value, performing a sparsification process on the adjusted model parameters of the at least one sub-model using a sparsification rule corresponding to the at least one sub-model to obtain updated model parameters of the at least one sub-model; The sub-models correspond to the sparsification rules one by one; The sparsification processing includes: when the number of steps in the current step is greater than the first preset value and less than or equal to the second preset value, multiplying the parameter value of the model parameter to be processed in the adjusted model parameters by a first preset coefficient, the first preset coefficient is a value greater than 0 and less than 1, and the second preset value is greater than the first preset value.

7. The device according to claim 6, wherein The processing module is further configured to perform one or more of the following: Adjusting the speed of the existing audio to obtain speed-adjusted audio, and using the speed-adjusted audio as the expanded audio; performing fundamental frequency adjustment on the existing audio to obtain audio after fundamental frequency adjustment, and using the audio after fundamental frequency adjustment as the expanded audio; The volume of the existing audio is adjusted to obtain a volume-adjusted audio sample, and the volume-adjusted audio sample is used as the expanded audio.

8. The device according to claim 6, wherein The training module is also used to: If the number of steps of the current step is less than or equal to a first preset value, the adjusted model parameters of the at least one sub-model are used as the updated model parameters of the at least one sub-model.

9. The device according to claim 6, wherein The training module is further configured to: Determine the model parameters to be processed in the adjusted model parameters by using the sparsification rule corresponding to the at least one sub-model; The parameter values ​​of the to-be-processed model parameters are multiplied by preset coefficients, and the parameter values ​​of the unprocessed model parameters other than the to-be-processed model parameters are kept unchanged to obtain updated model parameters of the at least one sub-model.

10. The device according to claim 9, wherein The training module is also used to: If the number of steps of the current step is greater than the second preset value, the parameter value of the model parameter to be processed is multiplied by a second preset coefficient, and the second preset coefficient is 0.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 5.

13. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images