Methods, apparatuses, devices, and media for managing vocoder models

Abstract

Methods, apparatuses, devices, and media for managing vocoder models are provided. In one method, a base model and a distilled model are respectively established based on a vocoder structure, the base model and the distilled model respectively describing a correlation between spectral features and sample point data in speech data, a data dimension of the base model being higher than a data dimension of the distilled model. The base model is trained based on training data to obtain a trained base model, the training data including sample spectral features and sample sample point data. Using the trained base model, a base prediction of the sample point data associated with the sample spectral features is determined. The distilled model is trained based on the sample spectral features and the base prediction of the sample point data to obtain a trained distilled model. In this way, the training of the distilled model can be guided by the base model, thereby reducing the learning difficulty and enabling the trained distilled model to reduce its own complexity while more realistically simulating natural language.

Description

Technical Field

[0001] The exemplary implementations of this disclosure generally relate to Text-to-Speech (TTS), and more specifically, to methods, apparatus, electronic devices, and storage media for managing vocoder models. Background Technology

[0002] In the field of speech synthesis technology, various text-to-speech conversion schemes have been developed. For example, vocoders can convert the features of text into speech signals that humans can perceive. Neural network technology has been widely applied in speech synthesis. For instance, vocoders based on various neural network models have been proposed, which can improve real-time performance while maintaining sound quality. However, high-quality vocoders typically employ complex neural network models, making it difficult to perform real-time speech generation on terminal devices with limited computing resources (e.g., mobile terminals). Therefore, how to reduce the complexity of vocoders and thus their resource overhead while maintaining sound quality has become a research hotspot. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for managing a vocoder model is provided. In this method, a base model and a distillation model are established based on the vocoder architecture. The base model and the distillation model describe the correlation between spectral features and sample point data in speech data, respectively, with the base model having a higher data dimensionality than the distillation model. The base model is trained using training data, including sample spectral features and sample sample point data, to obtain a trained base model. Using the trained base model, a base prediction of the sample point data associated with the sample spectral features is determined. Based on the base predictions of the sample spectral features and the sample point data, the distillation model is trained to obtain a trained distillation model.

[0004] In a second aspect of this disclosure, an apparatus for managing a vocoder model is provided. The apparatus includes: a setup module configured to set up a base model and a distillation model based on a vocoder structure, the base model and the distillation model respectively describing the correlation between spectral features and sample point data in speech data, the base model having a higher data dimensionality than the distillation model; a base model training module configured to train the base model based on training data including sample spectral features and sample sample point data to obtain a trained base model; a prediction module configured to use the trained base model to determine a base prediction of the sample point data associated with the sample spectral features; and a distillation model training module configured to train a distillation model based on the base prediction of the sample spectral features and the sample point data to obtain a trained distillation model.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to a first aspect of this disclosure when executed by the at least one processing unit.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method according to a first aspect of this disclosure.

[0007] It should be understood that the content described in this section is not intended to limit the key or essential features of the implementation of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0008] The above and other features, advantages, and aspects of various implementations of this disclosure will become more apparent in the following detailed description, taken in conjunction with the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A block diagram of an example environment in which the implementation of this disclosure can be implemented is shown;

[0010] Figure 2 A block diagram is shown illustrating a process for managing a vocoder model according to some implementations of this disclosure;

[0011] Figure 3 A block diagram is shown illustrating a process for generating a base model and a distillation model based on a vocoder architecture, according to some implementations of this disclosure.

[0012] Figure 4 A block diagram of a vocoder structure according to some implementations of this disclosure is shown;

[0013] Figure 5 A block diagram is shown illustrating the process for a parameter matrix in a compressed vocoder structure according to some implementations of this disclosure;

[0014] Figure 6 A block diagram is shown illustrating a process for generating speech data based on a trained distillation model, according to some implementations of this disclosure.

[0015] Figure 7 A flowchart is shown of a method for managing a vocoder model according to some implementations of this disclosure;

[0016] Figure 8A block diagram of an apparatus for managing a vocoder model according to some implementations of this disclosure is shown; and

[0017] Figure 9 A block diagram of a device capable of implementing various implementations of the present disclosure is shown. Detailed Implementation

[0018] Implementations of this disclosure will now be described in more detail with reference to the accompanying drawings. While some implementations of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the implementations set forth herein. Rather, these implementations are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and implementations of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0019] In the description of the implementation methods disclosed herein, the term "comprising" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one implementation" or "the implementation" should be understood as "at least one implementation". The term "some implementations" should be understood as "at least some implementations". Other explicit and implicit definitions may also be included below. As used herein, the term "model" can represent the relationships between various data. For example, the aforementioned relationships can be obtained based on various currently known and / or future-developed technical solutions.

[0020] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0021] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure through appropriate means in accordance with relevant laws and regulations, and user authorization should be obtained.

[0022] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0023] As an optional but non-restrictive implementation, in response to a user's active request, a prompt message can be sent to the user, for example, via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose whether to "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0025] Example Environment

[0026] In the context of this disclosure, speech synthesis can be achieved based on machine learning techniques. Specifically, the TTS process generally involves a text front-end and a model back-end. The back-end may include an acoustic model and a vocoder. Specifically, the acoustic model can generate corresponding acoustic spectral features based on the input text information, and the vocoder can output corresponding audio waveforms based on the input acoustic spectral features. See below for more details. Figure 1 Describe the process of implementing a vocoder model based on machine learning techniques.

[0027] Figure 1 A block diagram of an example environment 100 in which the implementation of this disclosure can be implemented is shown. Figure 1 In environment 100, it is desired to train and use a model (i.e., vocoder model 130) configured to generate corresponding sampling point data based on spectral features. That is, the model... Figure 1 As shown, environment 100 includes model training system 150 and model application system 152. Figure 1 The upper part illustrates the model training phase, and the lower part illustrates the model application phase. Before training, the parameters of vocoder model 130 can have initial values ​​or pre-trained parameter values ​​obtained through a pre-training process. During the training process, the parameter values ​​of vocoder model 130 can be updated and adjusted. After training is complete, a trained vocoder model 130' is obtained. At this point, the parameter values ​​of vocoder model 130' have been updated, and based on the updated parameter values, vocoder model 130' can be used in the model application phase to achieve the task of converting spectral features into corresponding sampling point data.

[0028] During the model training phase, a vocoder model 130 can be trained using a training dataset 110 comprising multiple training data points 112 and a model training system 150. Each training data point 112 may involve a binary format and includes sample spectral features 120 and sample sampling point data 122. The vocoder model 130 can then be trained using the training data 112, which includes the sample spectral features 120 and sample sampling point data 122. Specifically, the training process can be performed iteratively using a large amount of training data. After training is complete, the vocoder model 130 may include knowledge about speech synthesis. During the model application phase, the vocoder model 130' (which now has the trained parameter values) can be invoked using a model application system 152. For example, input data 140 (including spectral features 142 to be processed) can be received, and a prediction 144 of the audio corresponding to the spectral features 142 can be output.

[0029] exist Figure 1 In this context, the model training system 150 and the model application system 152 can include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. Terminal devices can involve any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, or any combination thereof, including accessories and peripherals of these devices or any combination thereof. Servers include, but are not limited to, mainframes, edge computing nodes, computing devices in cloud environments, etc.

[0030] It should be understood that Figure 1 The components and arrangements shown in environment 100 are merely examples, and a computing system suitable for implementing the exemplary implementations described in this disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown as separate, model training system 150 and model application system 152 may be integrated in the same system or device. Implementations of this disclosure are not limited in this respect. Exemplary implementations of model training and model application will continue to be described below with reference to the accompanying drawings.

[0031] Machine learning-based vocoder solutions have been proposed. The performance of the vocoder directly determines the listening experience of the synthesized audio (accuracy, clarity, etc.). Typically, vocoder models are highly complex and computationally intensive, limiting their deployment to devices with high computing resources. While smaller vocoder models have been developed for deployment on resource-constrained devices such as mobile terminals, their performance remains unsatisfactory. Therefore, generating vocoders more efficiently and converting spectral features into sampling point data has become a challenging and crucial aspect of speech synthesis.

[0032] Architecture of the vocoder model

[0033] Within the context of this disclosure, a knowledge distillation-based architecture is proposed for training vocoder models for deployment on end devices. See first... Figure 2 This describes a summary of an exemplary implementation according to the present disclosure. Figure 2 A block diagram of a process 200 for managing a vocoder model according to some implementations of this disclosure is shown. Figure 2 As shown by arrows 252 and 254, a base model 210 and a distillation model 220 can be generated based on the vocoder structure 250, respectively. In the context of this disclosure, the base model 210 can be a teacher model in model knowledge distillation, and the distillation model 220 can be a student model in model knowledge distillation. Specifically, the base model 210 and the distillation model 220 can describe the correlation between spectral features and sampling point data in speech data with different levels of precision. Here, the data dimension of the base model 210 can be higher than that of the distillation model 220. In other words, both the base model 210 and the distillation model 220 are vocoder models, and the base model 210 has higher precision.

[0034] It can be based on training data 112 (which may include, for example, training data 112). Figure 1 The sample spectral features 120 and sample sampling point data 122 shown are used to train the base model 210, thereby obtaining the trained base model 210. Further, as shown by arrow 242, the sample spectral features 120 in the training data 112 can be input into the trained base model 210 to determine the base prediction 232 of the sampling point data associated with the sample spectral features 120 via the inference process shown by arrow 244. Then, new training data 230 can be constructed based on the base prediction 232 of the sampling point data and the sample spectral features 120 in the original training data 112. As shown by arrow 246, the training data 230 can be used to train the distillation model 220.

[0035] Will understand, although Figure 2The process of training the distillation model 220 using one training data set 230 is only illustrative. Similar processing can be performed on each training data set 112 in the training dataset 110 to obtain multiple corresponding training data sets. The training of the distillation model 220 can be performed iteratively using multiple training data sets until a trained distillation model 220 that meets predetermined conditions is obtained.

[0036] It will be understood that the base model 210 here has a high data dimensionality and can achieve high accuracy, thus the output base prediction 232 also has high accuracy. Compared to the ground truth sample data 122 in the training data 112, the base prediction 232 can fit the ground truth sample data 122 better, meaning the generated speech data can realistically simulate natural speech. On the other hand, the base prediction 232 includes fewer speech details (e.g., details that are difficult for most human hearing to perceive), meaning the complexity of the output sample data is much lower than that of the ground truth sample data 122. Using the exemplary implementation of this disclosure, the original sample spectral features 120 and the base prediction 232 can be used to train the distillation model 220. In this way, the base model 210 can be used to guide the training of the distillation model 220, thereby reducing the learning difficulty and enabling the trained distillation model 220 to realistically simulate natural language while reducing its own complexity.

[0037] Model training process

[0038] In the following text, further details for obtaining the trained distillation model 220 will be described with reference to the accompanying drawings. First, see... Figure 3 Describe how the initial base model 210 and distillation model 220 were determined. Figure 3 A block diagram is shown illustrating a process 300 for generating a base model 210 and a distillation model 220 based on a vocoder structure, according to some implementations of this disclosure. According to an exemplary implementation of this disclosure, the base model 210 and the distillation model 220 can be built based on the same vocoder structure 250. The vocoder structure 250 here can be an autoregressive vocoder structure. Figure 3 A linear prediction coefficient (LPC) vocoder structure LPCNet is shown. The vocoder structure 250 may include a frequency network 310, a sampling rate network 312, and an LPC module 314.

[0039] LPCNet combines Digital Signal Processing (DSP) and Neural Networks (NN) to synthesize high-quality speech in real time on ordinary computing devices. Compared to vocoders based solely on DSP, combining DSP and NN ensures both the speed of speech data generation and the improvement of speech data quality based on the learning capabilities of NNs. LPCNet can receive spectral features and output sample point data of the corresponding speech data. Feature extraction is typically performed frame-by-frame; for a single input spectral feature, LPCNet will generate information from multiple (e.g., 100) sample points. Assuming each sample point has 65,536 possible values, the number of parameters in LPCNet would be enormous, typically requiring megabytes (M) of parameters. This results in inference time far from meeting real-time requirements.

[0040] According to an exemplary implementation of this disclosure, a complex base model 210 is proposed to guide the generation of a simplified distillation model 220. Specifically, the base model 210 and the distillation model 220 with different data dimensions can be generated based on the same vocoder structure 250. The two models can have identical network structures, differing only in that the control parameters of each unit in the two network structures can have different dimensions. Typically, the standard dimension of the control parameters of LPCNet can be 256 dimensions. In this case, the dimension of the control parameters of the base model 210 can be higher than this standard dimension (e.g., 384 dimensions), and the dimension of the control parameters of the distillation model 220 can be equal to or lower than this standard dimension (e.g., 256 dimensions or lower). In this case, the complexity of the distillation model 220 will be reduced, thereby reducing the computational and storage resources called up during inference and reducing inference time.

[0041] See below. Figure 4 For more details describing LPCNet, the Figure 4 A block diagram of a vocoder structure 400 according to some implementations of this disclosure is shown. For example... Figure 4 As shown, the vocoder structure 400 can receive spectral features 410 and output sampling point data S at each time point. t Specifically, the frame rate network 312 may include convolutional units 420 and 422, a combination unit 424, and fully connected units 426 and 428. The frame rate network 312 provides the input f of the conditional vector to the sampling rate network 314, calculating f once per frame, and f remains constant within the time range of that frame. The LPC module 314 may include an LPC unit 430 and a prediction unit 432, used to calculate linear prediction parameters P associated with each time point from the input spectral features 410. tP is calculated once per frame. t And P t It remains unchanged within the time frame of that frame.

[0042] According to an exemplary implementation of this disclosure, spectral feature 410 can be represented using a Mel frequency spectrum. Mel frequencies are proposed based on the characteristics of human hearing and have a non-linear relationship with Hz frequencies. Mel frequency cepstral coefficients (MFCCs) are used to calculate Hz spectral features using this relationship; MFCCs have been widely used in speech recognition. Due to the non-linear relationship between Mel frequencies and Hz frequencies, the calculation accuracy of MFCCs decreases as the frequency increases. In applications, often only low-frequency MFCCs are used, while mid- and high-frequency MFCCs are discarded.

[0043] like Figure 4 As shown, the sampling rate network 314 may include multiple units: a splicing unit 440, a Gate Recurrent Unit (GRU)-A 442, a GRU-B 444, a dual fully connected unit 446, and a Softmax unit 448. Specifically, the splicing unit 440 may receive the output f from the fully connected unit 428 and the linear prediction parameters P from the prediction unit 432. t Sampling data from previous time points t-1 and activation e from previous time points of sampling unit 450 t-1 In the context of this disclosure, the base model 210 and the distillation model 220 can be implemented based on the currently known and / or future LPCNet models and variations thereof. Further details regarding the network structures of the base model 210 and the distillation model 220 will be omitted below.

[0044] According to an exemplary implementation of this disclosure, it can be based on, as follows Figure 4 The structures shown are used to implement the base model 210 and the distillation model 220, respectively. Both the base model 210 and the distillation model 220 are vocoder models and can describe the correlation between spectral features and sampling point data in speech data, respectively. The difference between the base model 210 and the distillation model 220 is that they have different data dimensions, and the data dimension of the base model 210 can be higher than that of the distillation model 220.

[0045] Furthermore, the base model 210 can be trained using the various training data 112 in the training dataset 110. Here, the training data 112 may include sample spectral features 120 and sample sampling point data 122, where the sample sampling point data may include information from multiple sampling points in the speech data. In the initial training phase, the control parameters in the base model 210 may have initial values. The sample spectral features 120 can be input into the base model 210, and this initial base model 210 is used to determine the predictions of the sampling point data associated with the sample spectral features 120. Furthermore, a loss function can be constructed based on the difference between the predictions of the sampling point data and the sample sampling point data 122 to update the control parameters of the base model 210.

[0046] According to an exemplary implementation of this disclosure, the training process can be iteratively performed using each training data 112 in the training dataset 110 to obtain a trained base model 210. After sufficient training and testing, the trained base model 210 can accurately describe the correlation between the spectral features of the speech data and the sampling point data. Sample spectral features 120 can be input into the trained base model 210 to obtain basic predictions of the sampling point data.

[0047] Using the exemplary implementation of this disclosure, the base model 210, having a high data dimensionality, can accurately output sampling point data corresponding to spectral features. It will be understood that the ground truth sampling point data 122 is collected from real speech data and may therefore include environmental noise data and redundant data that is not sensitive to human hearing. Compared to the original ground truth sampling point data 122, the base prediction can reduce the impact of this noise and redundant data and represent the audible data that is sensitive to human hearing in a more concise and hearing-recognition-friendly manner. In this way, using the base prediction as training data for the distillation model 220 helps simplify the problem solved by the distillation model 220, thereby reducing the complexity of the distillation model 220.

[0048] According to an exemplary implementation of this disclosure, new training data 230 can be constructed based on the sample spectral features 120 and the basic predictions 232 of the sampling point data to train the distillation model 220. Assuming the training dataset 110 includes 10,000 training data points, 10,000 new training data points 230 can be generated based on these 10,000 training data points. Furthermore, the distillation model 220 can be trained iteratively using each of the training data points 230. In this case, the new training data 230 may include the sample spectral features 120 and the basic predictions 232, where the basic predictions 232 may include multiple sampling point information generated by the base model 210.

[0049] In the initial training phase, the control parameters in distillation model 220 can have initial values. Sample spectral features 120 can be input into distillation model 220, and this initial distillation model 220 is used to determine the prediction of the sampling point data associated with the sample spectral features 120. Furthermore, a loss function can be constructed based on the difference between the prediction of the sampling point data and the base prediction 232 to update the control parameters of distillation model 220.

[0050] According to an exemplary implementation of this disclosure, the training process can be iteratively executed using newly generated training data 112 to obtain a trained distillation model 220. It will be understood that the base predictions 232 in the training data 230 of the distillation model 220 are not the original collected ground truth data, but rather simplified sample point data after preprocessing. This significantly reduces the processing difficulty of the distillation model 220. In this way, the complexity of the trained distillation model 220 can be greatly reduced, thus preventing excessive delays when calling the distillation model 220 on terminal devices with limited computing and storage resources. Furthermore, since the base predictions 232 include accurate sample point data corresponding to the sample spectral features, the distillation model 220 trained using the base predictions 232 does not significantly degrade the quality of the generated speech data.

[0051] Using the exemplary implementation of this disclosure, a high-performance, long-processing-time base model 210 is used to guide the training of a fast-paced distillation model 220. This reduces the task difficulty of the distillation model 220 and enables it to achieve essentially the same speech synthesis performance as the base model 210. In this case, the distillation model 220 can weaken audio features that do not significantly affect the listening experience and generate speech data in real time with a processing speed higher than that of the base model 210.

[0052] The process of generating and training the base model 210 and the distillation model 220 has been described above. It will be understood that although the base model 210 and the distillation model 220 are generated separately, the purpose of the base model 210 is to generate training data for the distillation model 220; the base model 210 is not actually invoked in practical use. Due to the limitations of computing and storage resources on the terminal device, and the fact that other applications running on the terminal device may consume the already limited computing and storage resources, it is desirable to further optimize the distillation model 220 to improve the speed at which the distillation model 220 is invoked on the terminal device.

[0053] According to an exemplary implementation of this disclosure, control parameters in distillation model 220 can be updated based on sparsity operations, thereby reducing the storage space of distillation model 220 and improving the speed of speech synthesis. See also... Figure 4The vocoder structure shown, the distillation model 220, can include three parts: a frame rate network, an LPC module, and a sampling rate network. Since the frame rate network and LPC module are only called once when processing each spectral feature, the computational load involved in these modules is relatively small, meaning they do not significantly affect the size or running speed of the distillation model 220. However, the sampling rate network needs to output a large number of sampling points for each input spectral feature; therefore, the sampling rate network will affect the size of the distillation model 220 and consequently its running speed.

[0054] According to an exemplary implementation of this disclosure, the individual units included in the distillation model 220 (e.g., individual units in the sampling rate network) can be further optimized. The distillation model 220 is determined by the control parameters of each network; specifically, each unit in the sampling rate network includes a corresponding parameter matrix. According to an exemplary implementation of this disclosure, the target parameter matrix to be optimized can be selected from multiple parameter matrices based on the complexity of each parameter matrix. Figure 4 As shown, GRU-A 442 and GRU-B 444 in the sampling rate network 314 involve a large amount of data and complex calculations. Therefore, GRU-A 442 and GRU-B 444 can be optimized primarily. Alternatively and / or additionally, other units in the distillation model 220 can be optimized to a lesser extent.

[0055] According to an exemplary implementation of this disclosure, GRU-A 442 and GRU-B 444 can be optimized based on sparsity operations. In the context of this disclosure, a sparse matrix is ​​a matrix in which the number of non-zero elements is much smaller than the total number of elements in the matrix. The density of the matrix can be defined as the ratio of the number of non-zero elements to the total number of elements in the matrix.

[0056] See Figure 5 Describe more information about sparsification operations. Figure 5 A block diagram 500 shows a parameter matrix 510 in a compressed vocoder structure according to some implementations of this disclosure. For example... Figure 5 As shown, the control parameters of GRU-A 442 may include, for example, a parameter matrix 510, which can store the network weights in GRU-A 442. During the training of the distillation model 220, the sparsity operation of the parameter matrix 510 can be implemented progressively. In other words, the values ​​in the parameter matrix 510 can be modified progressively during iterative training, and the distillation model 220 is trained in a direction that makes the parameter matrix 510 a sparse matrix.

[0057] Specifically, the parameter matrix 510 can be divided into groups of continuous matrix elements. For example, the length of the continuous matrix elements can be determined based on the distribution of zero-value elements and non-zero elements in the parameter matrix 510. For example, if there are many zero-value elements in the parameter matrix 510 and their distribution is relatively continuous, the length of the continuous matrix elements can be set to be relatively long (e.g., 16 or other values); if there are few zero-value elements in the parameter matrix 510 and their distribution is relatively scattered, the length of the continuous matrix elements can be set to be relatively short (e.g., 8 or other values).

[0058] According to an exemplary implementation of this disclosure, the sparsification process is described in units of 16 consecutive zero-value elements. In this case, a set of consecutive matrix elements 520 can represent 16*1 consecutive zero-value matrix elements. That is, the parameter matrix 510 of the GRU-A442 can be represented as multiple 16*1 small matrices. Assuming the dimension of the parameter matrix 510 is N*N (where N is an integer multiple of 16), the parameter matrix 510 can be represented as a matrix with dimension N*N. A small matrix.

[0059] According to an exemplary implementation of this disclosure, it is possible to determine whether each element in a small matrix satisfies a predetermined threshold condition. For example, the threshold condition can be defined based on the distances between the 16 original matrix elements in each element and the zero value. If the distance between each (or a predetermined proportion, e.g., more than 50%) of the original matrix elements and the zero value is less than 0.05 (or some other value), the threshold condition can be considered satisfied. As another example, the threshold condition can be defined based on the spatial distances between the 16 original matrix elements and the zero value, and so on. In this case, a parameter matrix can be stored using sparse identifiers (e.g., the zero value or other predetermined identifiers) and the positions of the zero-value elements within the small matrix.

[0060] In such Figure 5 In the example shown, if one matrix element is stored in a single storage unit, storing the parameter matrix 510 in the conventional manner would occupy N*N storage units. According to an exemplary implementation of this disclosure, the sparsified parameter matrix can be stored as a sparse matrix. Specifically, the non-zero parts of the parameter matrix 510 can be stored in the conventional manner, requiring N*N-16 storage units. For the zero-value part, two storage units are required (one unit stores the position of a group of consecutive matrix elements 520, and the other unit stores the value "0"). Compared to the conventional storage method, sparsifying the parameter matrix 510 and storing it using a sparse matrix significantly reduces the storage space occupied by the parameter matrix 510.

[0061] Will understand, although Figure 5The diagram only schematically illustrates the case where parameter matrix 510 comprises a continuous 16*1 matrix of zero-valued elements. According to an exemplary implementation of this disclosure, during the training of distillation model 220, if certain continuous matrix elements in parameter matrix 510 are found to satisfy a threshold condition, these continuous matrix elements can be set to zero. In subsequent iterative training processes, when adjusting parameter matrix 510 based on the loss function of distillation model 220, constraints can be set for parameter adjustments during training. For example, only the values ​​of the non-zero elements other than the set of continuous matrix elements set to zero can be adjusted. Using the exemplary implementation of this disclosure, the values ​​of the non-zero portions of parameter matrix 510 will "compensate" for the knowledge provided by the matrix elements that would otherwise need to be set to zero. Thus, the accuracy of distillation model 220 can be improved while reducing its size.

[0062] According to an exemplary implementation of this disclosure, from a storage perspective, a 16*1 sparsified matrix only needs to record the parameters of non-zero elements and the position information of all-zero elements during storage, thus significantly reducing storage space. For example, a matrix that has been 50% sparsified can reduce storage space by almost 50%. From an inference speed perspective, the matrix can be divided into multiple smaller matrices for computation, with the results of the all-zero element matrices directly set to 0, and finally the computation results of the multiple smaller matrices combined to form the computation result of the original parameter matrix 510. In this way, the optimized distillation model 220 can significantly improve inference speed and achieve real-time performance.

[0063] According to an exemplary implementation of this disclosure, using a 2*1 sparsified matrix as a simple example, the operations related to the sparsification matrix are described. Assuming the original matrix is ​​[1.1,2.2,0.1,-0.1,1.1,2.1,0.0,0.1], after 2*1 sparsification, the matrix can be represented as [1.1,2.2,0,0,1.1,2.1,0,0]. When storing the sparsified matrix, the storage method changes from storing the original 8 floating-point numbers to storing only 4 non-zero floating-point numbers: [1.1,2.2,1.1,2.1] and the position information of the 2*1 sparse identifier [3,6]. In this way, the storage space can be significantly reduced. When performing calculations based on a sparse matrix, such as [1.1,2.2,0,0,1.1,2.1,0,0]*2, we can directly calculate [1.1,2.2,1.1,2.1]*2, padding the 0.0 into the corresponding sparse identifier positions to obtain the final result: [2.2,4.4,0.0,0.0,2.2,4.2,0.0,0.0]. In this way, sparse matrices exhibit high performance in both storage and computation. It will be understood that the above only illustrates a simple operation related to sparse matrices; sparse matrices and sparse identifiers can have other dimensions, and other operations can be performed based on similar principles.

[0064] Model application process

[0065] The training process of distillation model 220 has been described above. The following section will describe how to use distillation model 220 to generate speech data. After sufficient training and testing, the trained distillation model 220 can accurately describe the correlation between the spectral features of speech data and the sampling point data. Target spectral features to be processed can be input into the trained distillation model 220 to obtain speech data corresponding to the target spectral features.

[0066] Figure 6 A block diagram 600 is shown for generating speech data based on a trained distillation model, according to some implementations of this disclosure. For example... Figure 6 As shown, a trained distillation model 220 can be deployed on the terminal device 610. This trained distillation model 220 can be used to receive spectral features and generate corresponding speech data. According to an exemplary implementation of this disclosure, an acoustic model 620 can also be deployed at the terminal device 610, which can receive text to be processed and generate corresponding spectral features.

[0067] According to an exemplary implementation of this disclosure, the distillation model 220 obtained based on the process described above can be applied to different environments. For example, in a navigation application, the navigation application can determine specific navigation instructions based on the current location of the terminal device 610 and surrounding road information. Assuming the navigation application outputs the navigation instruction text "Turn right at the next intersection," the acoustic model 620 can receive the text and the distillation model 220 can generate speech data reading "Turn right at the next intersection." As another example, in a reading application, the reading application can receive the original text that is expected to be converted into speech data; for example, the reading application can receive the text "The weather is sunny today" and generate corresponding speech data. Using the exemplary implementation of this disclosure, the speech generation capability of the distillation model 220 can be invoked in different application environments to generate speech data in real-time or near real-time.

[0068] Example process

[0069] Figure 7 A flowchart illustrating a method for managing a vocoder model according to some implementations of this disclosure is shown. Specifically, at box 710, a base model and a distillation model are established based on the vocoder structure, respectively. The base model and the distillation model describe the correlation between spectral features and sample point data in the speech data, respectively, with the base model having a higher data dimension than the distillation model. At box 720, the base model is trained based on training data, including sample spectral features and sample sample point data, to obtain a trained base model. At box 730, using the trained base model, a base prediction of the sample point data associated with the sample spectral features is determined. At box 740, based on the base predictions of the sample spectral features and sample point data, the distillation model is trained to obtain a trained distillation model.

[0070] According to an exemplary implementation of this disclosure, establishing a base model and a distillation model respectively includes: establishing a base model and a distillation model based on the same autoregressive vocoder structure, wherein the data dimension of the base model is higher than the standard data dimension of the autoregressive vocoder structure, and the data dimension of the distillation model is not higher than the standard data dimension of the autoregressive vocoder structure.

[0071] According to an exemplary implementation of this disclosure, the sampling point data includes multiple sampling point information in the speech data, and training a base model based on training data includes: using the base model to determine the prediction of the sampling point data associated with the sample spectral features; and updating the control parameters of the base model based on the difference between the prediction of the sampling point data and the sample sampling point data to obtain the trained base model.

[0072] According to an exemplary implementation of this disclosure, training a distillation model includes: using the distillation model to determine predictions of sampling point data associated with sample spectral features; and updating control parameters of the distillation model based on the difference between the predictions of the sampling point data and the baseline predictions of the sampling point data to obtain a trained distillation model.

[0073] According to an exemplary implementation of this disclosure, the distillation model includes at least one gated loop unit, and the control parameters of the distillation model include a plurality of parameter matrices of the gated loop unit in the at least one gated loop unit.

[0074] According to an exemplary implementation of this disclosure, training a distillation model includes: selecting a target parameter matrix from multiple parameter matrices based on the complexity of the multiple parameter matrices; setting a set of continuous matrix elements to zero in response to determining that a set of continuous matrix elements in the target parameter matrix satisfies a threshold condition; and adjusting the portion of the target parameter matrix other than the set of continuous matrix elements based on the difference.

[0075] According to an exemplary implementation of this disclosure, it further includes: determining the length of a set of continuous matrix elements based on the distribution of zero-value matrix elements in the target parameter matrix.

[0076] According to an exemplary implementation of this disclosure, it further includes: determining the positions of a set of continuous matrix elements in the target parameter matrix; and representing the target parameter matrix as a sparse matrix based on the positions and sparse identifiers.

[0077] According to one exemplary implementation of this disclosure, the autoregressive vocoder structure includes an autoregressive vocoder structure based on linear prediction coefficients, and the spectral features include Mel spectral features.

[0078] According to an exemplary implementation of this disclosure, it further includes: inputting target spectral features into a trained distillation model to obtain target sampling point data associated with the target spectral features.

[0079] Example devices and equipment

[0080] Figure 8A block diagram of an apparatus for managing a vocoder model according to some implementations of the present disclosure is shown. The apparatus 800 includes: a setup module 810 configured to set up a base model and a distillation model based on the vocoder structure, the base model and the distillation model describing the correlation between spectral features and sample point data in speech data, respectively, the base model having a higher data dimension than the distillation model; a base model training module 820 configured to train the base model based on training data including sample spectral features and sample sample point data to obtain a trained base model; a prediction module 830 configured to use the trained base model to determine a base prediction of the sample point data associated with the sample spectral features; and a distillation model training module 840 configured to train the distillation model based on the base prediction of the sample spectral features and the sample point data to obtain a trained distillation model.

[0081] According to an exemplary implementation of this disclosure, the model building module 810 includes: a model building module configured to build a base model and a distillation model respectively based on the same autoregressive vocoder structure, wherein the data dimension of the base model is higher than the standard data dimension of the autoregressive vocoder structure, and the data dimension of the distillation model is not higher than the standard data dimension of the autoregressive vocoder structure.

[0082] According to an exemplary implementation of this disclosure, the sampling point data includes multiple sampling point information in the speech data, and the basic model training module 820 includes: a sampling point prediction module configured to use the basic model to determine the prediction of the sampling point data associated with the sample spectral features; and an update module configured to update the control parameters of the basic model based on the difference between the prediction of the sampling point data and the sample sampling point data to obtain a trained basic model.

[0083] According to an exemplary implementation of this disclosure, the distillation model training module 840 includes: a sampling point prediction module configured to use the distillation model to determine predictions of sampling point data associated with sample spectral features; and an update module configured to update control parameters of the distillation model based on the difference between the predictions of the sampling point data and the baseline predictions of the sampling point data, to obtain a trained distillation model.

[0084] According to an exemplary implementation of this disclosure, the distillation model includes at least one gated loop unit, and the control parameters of the distillation model include a plurality of parameter matrices of the gated loop unit in the at least one gated loop unit.

[0085] According to an exemplary implementation of this disclosure, the distillation model training module 840 includes: a selection module configured to select a target parameter matrix from multiple parameter matrices based on the complexity of the multiple parameter matrices; a setting module configured to set a set of continuous matrix elements to zero in response to determining that a set of continuous matrix elements in the target parameter matrix satisfies a threshold condition; and an adjustment module configured to adjust the portion of the target parameter matrix other than the set of continuous matrix elements based on differences.

[0086] According to an exemplary implementation of this disclosure, the apparatus 800 further includes: a length determination module configured to determine the length of a set of continuous matrix elements based on the distribution of zero-value matrix elements in the target parameter matrix.

[0087] According to an exemplary implementation of this disclosure, the apparatus 800 further includes: a position determination module configured to determine the positions of a set of continuous matrix elements in a target parameter matrix; and a representation module configured to represent the target parameter matrix as a sparse matrix based on the positions and sparse identifiers.

[0088] According to one exemplary implementation of this disclosure, the autoregressive vocoder structure includes an autoregressive vocoder structure based on linear prediction coefficients, and the spectral features include Mel spectral features. According to one exemplary implementation of this disclosure, the apparatus 800 further includes: a target prediction module configured to input target spectral features into a trained distillation model to obtain target sampling point data associated with the target spectral features.

[0089] Figure 9 A block diagram of a device 900 capable of implementing various implementations of the present disclosure is shown. It should be understood that... Figure 9 The computing device 900 shown is merely exemplary and should not be construed as limiting the functionality and scope of the implementation described herein. Figure 9 The computing device 900 shown can be used to implement the method described above.

[0090] like Figure 9 As shown, computing device 900 is in the form of a general-purpose computing device. Components of computing device 900 may include, but are not limited to, one or more processors or processing units 910, memory 920, storage devices 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. Processing unit 910 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 920. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of computing device 900.

[0091] Computing device 900 typically includes multiple computer storage media. Such media can be any available media accessible to computing device 900, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 920 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 930 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media capable of storing information and / or data (e.g., training data for training) and accessible within computing device 900.

[0092] The computing device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 9 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 920 may include computer program product 925 having one or more program modules configured to perform various methods or actions of various implementations of this disclosure.

[0093] The communication unit 940 enables communication with other computing devices via a communication medium. Additionally, the components of the computing device 900 can function as a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the computing device 900 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0094] Input device 950 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 960 can be one or more output devices, such as a monitor, speaker, printer, etc. Computing device 900 can also communicate as needed with one or more external devices (not shown) via communication unit 940. These external devices, such as storage devices, display devices, etc., can communicate with one or more devices that enable user interaction with computing device 900, or with any device (e.g., network card, modem, etc.) that enables computing device 900 to communicate with one or more other computing devices. Such communication can be performed via input / output (I / O) interfaces (not shown).

[0095] According to exemplary implementations of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is provided that stores a computer program thereon, which, when executed by a processor, implements the methods described above.

[0096] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0097] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0098] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0100] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for managing a vocoder model, comprising: Based on the data dimension of the vocoder structure, the first data dimension of the base model and the second data dimension of the distillation model are determined respectively. The data dimension represents the precision of the vocoder structure, and the first data dimension of the base model is higher than the second data dimension of the distillation model. Based on the vocoder structure, the base model is established based on the first data dimension, and the distillation model is established based on the second data dimension. The base model and the distillation model have the same network structure and respectively describe the correlation between spectral features and sampling point data in speech data. The base model is trained based on training data to obtain a trained base model, wherein the training data includes sample spectral features and sample sampling point data; Using the trained base model, determine the basic predictions of the sampling point data associated with the sample spectral features; as well as Based on the sample spectral features and the basic predictions of the sampling point data, the distillation model is trained to obtain the trained distillation model.

2. The method according to claim 1, wherein the vocoder structure includes an autoregressive vocoder structure, and establishing the base model and the distillation model includes: The base model and the distillation model are established based on the same autoregressive vocoder structure. The first data dimension of the base model is higher than the standard data dimension of the autoregressive vocoder structure, and the second data dimension of the distillation model is not higher than the standard data dimension of the autoregressive vocoder structure.

3. The method according to claim 2, wherein the sampling point data includes multiple sampling point information in the speech data, and training the base model based on the training data includes: Using the aforementioned basic model, predictions of sampling point data associated with the sample spectral features are determined; as well as Based on the difference between the prediction of the sampling point data and the sample sampling point data, the control parameters of the base model are updated to obtain the trained base model.

4. The method of claim 1, wherein training the distillation model comprises: Using the distillation model, predictions of sampling point data associated with the sample spectral features are determined; as well as Based on the difference between the prediction of the sampled data and the basic prediction of the sampled data, the control parameters of the distillation model are updated to obtain the trained distillation model.

5. The method of claim 4, wherein the distillation model comprises at least one gated loop unit, and the control parameters of the distillation model comprise a plurality of parameter matrices of the gated loop unit in the at least one gated loop unit.

6. The method of claim 5, wherein training the distillation model comprises: Based on the complexity of the multiple parameter matrices, a target parameter matrix is ​​selected from the multiple parameter matrices; In response to determining that a set of consecutive matrix elements in the target parameter matrix satisfies a threshold condition, the set of consecutive matrix elements is set to zero; as well as Based on the difference, adjust the portion of the target parameter matrix other than the set of continuous matrix elements.

7. The method of claim 6, further comprising: The length of the set of continuous matrix elements is determined based on the distribution of zero-value matrix elements in the target parameter matrix.

8. The method of claim 6, further comprising: Determine the positions of the set of consecutive matrix elements in the target parameter matrix; as well as The target parameter matrix is ​​represented as a sparse matrix based on the location and sparse identifier.

9. The method of claim 2, wherein the autoregressive vocoder structure comprises an autoregressive vocoder structure based on linear prediction coefficients, and the spectral features comprise Mel spectral features.

10. The method of claim 1, further comprising: The target spectral features are input into the trained distillation model to obtain target sampling point data associated with the target spectral features.

11. An apparatus for managing a vocoder model, comprising: A determination module is configured to determine a first data dimension of the base model and a second data dimension of the distillation model based on the data dimension of the vocoder structure, wherein the data dimension represents the precision of the vocoder structure, and the first data dimension of the base model is higher than the second data dimension of the distillation model. A module is configured to build the base model based on the first data dimension and the distillation model based on the second data dimension, according to the vocoder structure. The base model and the distillation model have the same network structure and respectively describe the correlation between spectral features and sampling point data in speech data. A base model training module is configured to train the base model based on training data to obtain a trained base model, the training data including sample spectral features and sample sampling point data; The prediction module is configured to use the trained base model to determine a base prediction of the sampling point data associated with the sample spectral features; as well as A distillation model training module is configured to train the distillation model based on the sample spectral features and the sampling point data to obtain the trained distillation model.

12. An electronic device, comprising: At least one processing unit; as well as At least one memory, coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 10 when executed by the at least one processing unit.

13. A computer-readable storage medium having a computer program stored thereon, the computer program causing the processor to implement the method according to any one of claims 1 to 10 when executed by a processor.

Images