Method for training a multitask processing model, method for speech synthesis, and related products

Abstract

This invention provides a method for training a multi-task processing model and related products. The method includes acquiring training data for the multi-task processing model and multiple teacher models, wherein the multi-task processing model is configured to perform multiple text analysis tasks in speech synthesis, and each teacher model is configured to perform a single text analysis task; determining a target loss for the multi-task processing model based on the prediction results of the teacher models and the multi-task processing model on the training data; and optimizing the training of the multi-task processing model based on the target loss. Through the technical solution of this invention, the multi-task processing model can be trained to be applied to multiple tasks simultaneously, breaking the traditional pipelined combination analysis method and achieving a balance between quality and performance. In addition, this invention also provides a method for speech synthesis.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of speech processing technology, and more specifically, to methods for training multi-task processing models, methods for speech synthesis, and electronic devices and computer-readable storage media for performing the aforementioned methods. Background Technology

[0002] This section is intended to provide background or context for embodiments of the invention set forth in the claims. The description herein may include concepts that may be explored, but not necessarily concepts that have been previously conceived or explored. Therefore, unless otherwise stated, what is described in this section is not prior art for the purposes of this application's specification and claims, and is not acknowledged as prior art simply by virtue of its inclusion in this section.

[0003] The essence of speech synthesis technology can be understood as converting text into speech and enabling machines to speak. In related technologies, a speech synthesis system can be composed of two parts: a front-end and a back-end. The front-end performs various text analyses on the input text to generate a linguistic specification, which the back-end then uses to generate the corresponding speech. Therefore, the front-end can be seen as a linguist, the most complex part of the entire speech synthesis system. Generating this linguistic specification requires many steps, and errors in any module will directly affect the final speech synthesis result. However, currently, the front-end primarily uses a pipelined analysis approach for text analysis tasks, meaning only one text analysis task is executed at a time, with the result of the previous task serving as the input for the next. In this traditional pipelined analysis method, each module consumes independent resources, resulting in significant resource consumption. Furthermore, cascading errors between modules are common, leading to high maintenance costs. It can be seen that this traditional streaming-line analysis method has relatively limited application scenarios. It requires high-performance platform support to ensure the quality of text analysis. It is difficult to achieve a balance between quality and performance on platforms or systems with limited performance (especially embedded systems). Summary of the Invention

[0004] In known speech synthesis technologies, the front-end analysis of text is not ideal, which is a very frustrating process.

[0005] Therefore, there is a great need for an improved scheme for training multi-task processing models that can support the analysis and processing of multiple text analysis tasks in speech synthesis, thereby achieving a balance between quality and performance.

[0006] In this context, embodiments of the present invention aim to provide a method for training a multi-task processing model and related products.

[0007] Furthermore, embodiments of the present invention also provide a method and related products for speech synthesis based on the above-described multi-task processing model.

[0008] In a first aspect of the present invention, a method for training a multi-task processing model is proposed, comprising: acquiring training data for the multi-task processing model and multiple teacher models, wherein the multi-task processing model is configured to perform multiple text analysis tasks in speech synthesis, and each teacher model is configured to perform a single type of text analysis task; determining a target loss for the multi-task processing model based on the prediction results of the teacher models and the multi-task processing model for the training data; and optimizing the training of the multi-task processing model based on the target loss.

[0009] In one embodiment of the present invention, the training data includes unlabeled data and mixed-label data, wherein the data in the mixed-label data is labeled with at least some task labels. Determining the target loss for the multi-task processing model includes: determining a first target loss based on the prediction results of each of the teacher models and the multi-task processing model for the unlabeled data in response to the training data containing the mixed-label data; and determining a second target loss based on the prediction results of some of the teacher models and the multi-task processing model for the mixed-label data in response to the training data containing the mixed-label data.

[0010] In another embodiment of the present invention, determining the first target loss based on the prediction results of each teacher model and the multi-task processing model for the unlabeled data includes: constructing a first distillation loss based on the prediction results of each teacher model and the multi-task processing model for the unlabeled data, and determining the first target loss based on the first distillation loss, so as to guide the multi-task processing model to learn each teacher model based on the first distillation loss.

[0011] In another embodiment of the present invention, determining the second target loss based on the prediction results of the mixed-label data by the teacher model and the multi-task processing model, and the labeled task labels, includes: inputting the partially labeled data into the multi-task processing model and the teacher model for performing the task corresponding to the missing data label, respectively, for prediction, and constructing a second distillation loss based on the prediction results; and calculating the cross-entropy loss between the prediction results of the multi-task processing model for the mixed-label data and the corresponding labeled task labels, and determining the second target loss based on the second distillation loss and the cross-entropy loss.

[0012] In another embodiment of the present invention, optimizing the multi-task processing model based on the target loss includes: optimizing the multi-task processing model using the teacher model and the first target loss determined by the multi-task processing model for the unlabeled data and the second target loss determined by the multi-task processing model for the mixed labeled data.

[0013] In one embodiment of the present invention, obtaining multiple teacher models for the multi-task processing model includes: obtaining labeled training data for each type of text analysis task; and fine-tuning the base language model using the labeled training data for each type of text analysis task to obtain a corresponding teacher model for performing each type of text analysis task.

[0014] In another embodiment of the present invention, the multi-type text analysis task includes a polyphonic character analysis task, a word segmentation analysis task, and a prosodic analysis task, and the basic language model includes the BERT-Base model.

[0015] In another embodiment of the present invention, the training data further includes multi-task data, wherein the data in the multi-task data is labeled with the labels of all tasks, and the method further includes: fine-tuning the multi-task processing model using the multi-task data.

[0016] In another embodiment of the present invention, wherein the multitasking model is applied to an electronic device, the method further includes: obtaining the resource configuration of the electronic device; and optimizing the multitasking model according to the resource configuration.

[0017] In one embodiment of the present invention, optimizing the multi-task processing model according to the resource configuration includes: adjusting the model architecture of the teacher model and / or the multi-task processing model according to the resource configuration; and / or adjusting the number of tasks and / or task execution priority of the multi-type text analysis tasks according to the resource configuration.

[0018] In a second aspect of the present invention, a speech synthesis method is proposed, comprising: acquiring a trained multi-task processing model, wherein the multi-task processing model is trained by the method described above; performing multi-task text analysis processing on the acquired text using the multi-task processing model to obtain a prediction result; and determining a speech synthesis result of the text based on the prediction result of the multi-task processing model.

[0019] In a third aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory storing computer instructions for training a multitasking model or computer instructions for speech synthesis, wherein when the computer instructions are executed by the processor, the electronic device causes the electronic device to perform the methods described in accordance with the foregoing and the following embodiments.

[0020] In a fourth aspect of the present invention, a computer-readable storage medium is provided, comprising program instructions for training a multi-task processing model or program instructions for speech synthesis, wherein when the program instructions are executed by a processor, the method described according to the foregoing and the following embodiments is implemented.

[0021] The method and related products for training a multi-task processing model according to embodiments of the present invention cleverly train a multi-task processing model by combining training data and multiple teacher models supporting a single type of text analysis task, so as to realize the processing of multiple types of text analysis tasks in speech synthesis based on the trained multi-task processing model. Therefore, the solution of this application, by training a multi-task processing model so that it can be applied to multiple tasks simultaneously, breaks the traditional pipelined combinatorial analysis method. This not only saves resources such as text analysis time and space, but also the architecture of the multi-task processing model, which can simultaneously realize multi-task analysis, is much simpler and easier to maintain than the pipelined combinatorial system architecture. Furthermore, the multi-task processing model, which can effectively save resources and is easy to maintain, is more universal, especially suitable for some resource-constrained scenarios with high requirements for speech quality, thus better facilitating a balance between performance and quality.

[0022] Furthermore, in some embodiments of the present invention, the training of the multi-task processing model does not rely on the difficult-to-obtain multi-task labeled data. Instead, it cleverly utilizes readily available unlabeled data and mixed-label data combined with semi-supervised distillation techniques to train the multi-task processing model. This not only solves the problem of the difficulty in obtaining multi-task data in the multi-task processing model, but also enables the multi-task processing model to effectively learn the capabilities of the teacher model through distillation based on a large amount of unlabeled data. At the same time, the integration of mixed-label data learning allows the multi-task processing model to learn in conjunction with actual labeled data during the distillation learning process, ensuring that the final model does not exhibit large deviations and is more stable.

[0023] Furthermore, in other embodiments of the present invention, the multi-tasking model can also be optimized and adjusted according to the specific resource configuration (e.g., memory, processing performance, device basic configuration, etc.) of the applied electronic device, so that the multi-tasking model can be more adapted to the electronic device (especially for some resource-constrained devices such as dictionary pens), so as to better balance performance and quality. Attached Figure Description

[0024] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0025] Figure 1 A block diagram schematically illustrates an exemplary computing system 100 suitable for implementing embodiments of the present invention; Figure 2 A schematic flowchart illustrating a method for training a multi-task processing model according to an embodiment of the present invention is shown. Figure 3 A schematic flowchart of a method for training a multi-task processing model according to another embodiment of the present invention is shown. Figure 4 A schematic flowchart of a method for training a multi-task processing model according to another embodiment of the present invention is shown. Figure 5 A schematic diagram of the structure of a training multi-task processing model according to yet another embodiment of the present invention is shown. Figure 6 A schematic diagram illustrating the structure of a speech synthesis method according to an embodiment of the present invention is shown; and Figure 7 A schematic diagram of the structure of an electronic device according to an embodiment of the present invention is shown.

[0026] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0027] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0028] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0029] Figure 1 A block diagram of an exemplary computing system 100 suitable for implementing embodiments of the present invention is shown. For example... Figure 1 As shown, the computing system 100 may include: a central processing unit (CPU) 101, random access memory (RAM) 102, read-only memory (ROM) 103, a system bus 104, a hard disk controller 105, a keyboard controller 106, a serial interface controller 107, a parallel interface controller 108, a display controller 109, a hard disk 110, a keyboard 111, a serial external device 112, a parallel external device 113, and a display 114. Among these devices, the CPU 101, RAM 102, ROM 103, hard disk controller 105, keyboard controller 106, serial controller 107, parallel controller 108, and display controller 109 are coupled to the system bus 104. The hard disk 110 is coupled to the hard disk controller 105, the keyboard 111 is coupled to the keyboard controller 106, the serial external device 112 is coupled to the serial interface controller 107, the parallel external device 113 is coupled to the parallel interface controller 108, and the display 114 is coupled to the display controller 109. It should be understood that... Figure 1 The structural block diagrams described are for illustrative purposes only and are not intended to limit the scope of the invention. In some cases, certain devices may be added or removed depending on the specific circumstances.

[0030] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, method, or computer program product. Therefore, this disclosure can be specifically implemented as entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," "unit," or "system." Furthermore, in some embodiments, the present invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0031] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (not exhaustive) of a computer-readable storage medium may include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0032] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device.

[0033] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0034] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0035] Embodiments of the present invention will now be described with reference to flowchart illustrations and block diagrams of apparatus (or systems) of embodiments thereof. It should be understood that each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine that, when executed by the computer or other programmable data processing apparatus, creates means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0036] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce a product comprising an instruction apparatus that implements the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0037] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable apparatus can provide a process for implementing the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0038] According to embodiments of the present invention, a method for training a multi-task processing model and a method for speech synthesis, as well as related products, are proposed. Furthermore, any number of elements in the accompanying drawings is for illustrative purposes only and not as a limitation, and any naming is for distinguishing purposes only and has no limiting meaning.

[0039] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments. Invention Overview The inventors have discovered that the text analysis performance of the front-end in existing speech synthesis technologies is unsatisfactory. Specifically, in traditional speech synthesis technologies, the front-end text analysis often employs a pipelined approach, meaning that only one text analysis task is executed at a time, with the result of the previous task serving as the input for the next. For example, this pipelined approach for tasks such as word segmentation and part-of-speech tagging first performs word segmentation, and then uses the result of word segmentation as the input for part-of-speech tagging. Consequently, each module used for tasks such as word segmentation and part-of-speech tagging not only requires independent resources (e.g., each module needs its own space and additional time to wait for the processing of the previous task), resulting in significant resource consumption. Furthermore, if the output of the module processing the previous task is incorrect, it will inevitably affect the analysis of the next task, leading to numerous cascading errors between models and high maintenance costs. This limits the application scenarios of this approach, especially in embedded systems where it is difficult to achieve a balance between quality and performance.

[0041] Based on this, the inventors discovered through research that a multi-task processing model that can be applied to multiple analysis tasks in speech synthesis can be trained. This multi-task processing model can save resources such as text analysis time and space, and effectively simplify the architecture of the text analysis system.

[0042] Furthermore, a well-trained multi-task processing model is used to achieve speech synthesis, making the entire speech synthesis technology more universal and achieving a better balance between performance and quality. In particular, it enables the speech synthesis system to synthesize high-quality speech on resource-limited devices such as dictionary pens.

[0043] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below.

[0044] Exemplary methods The following is for reference. Figure 2 This section describes a method for training a multi-task processing model according to an exemplary embodiment of the present invention. It should be noted that the embodiments of the present invention can be applied to any applicable scenario.

[0045] Figure 2 A method 200 for training a multi-task processing model according to an embodiment of the present invention is illustrated schematically.

[0046] like Figure 2 As shown, in step S201, training data for the multi-task processing model and multiple teacher models can be obtained. It is understood that this multi-task processing model can be configured to perform multiple text analysis tasks in speech synthesis. For example, speech synthesis may involve tasks such as word segmentation, part-of-speech tagging, polyphonic character analysis, and prosodic analysis, thus the multi-task processing model supports simultaneous application to these multiple tasks. The teacher model can generally be a single network or a collection of networks, possessing good performance and generalization ability. Here, the teacher model can be configured to perform only a single type of text analysis task (e.g., any one of the tasks such as word segmentation, part-of-speech tagging, polyphonic character analysis, and prosodic analysis). Preferably, for each type of text analysis task in speech synthesis, one teacher model performs one type of text analysis task. It should be noted that the number of text analysis tasks involved in speech synthesis is not limited here; the task types listed above are merely illustrative examples.

[0047] Furthermore, the aforementioned training data can be understood as data that can be used to train a multi-task processing model. In some embodiments, this training data can be easily obtained unlabeled data or mixed-label data with missing partial labels. For example, some text corpora can be used as training data; if they have no labels, they can be understood as unlabeled data. If the text corpus has one or more of the multi-task labels such as word segmentation labels or polyphonic character labels, the training data can be understood as mixed-label data. Of course, the aforementioned training data can also include some data with complete task labels that are more difficult to obtain. In this application, unlabeled data and mixed-label data are preferably used. Using these relatively easy-to-obtain data as training data can effectively reduce the difficulty of model training.

[0048] Next, in step S202, the target loss for the multi-task processing model can be determined based on the prediction results of the aforementioned teacher model and multi-task processing model on the training data. Those skilled in the art have a certain mindset regarding the training of multi-task processing models, typically believing that a large amount of fully labeled data can be used to train the model. However, since obtaining a large amount of effective fully labeled data requires significant human and material resources, it is difficult to acquire such data, leading to a continued reliance on pipeline-style analysis methods. The solution in this application breaks through this mindset by allowing the teacher model, capable of performing a single-class text analysis task, to participate in the training process of the multi-task processing model. Specifically, the teacher model is involved in the training by utilizing the prediction results of both the teacher model and the multi-task processing model to determine the target loss. This allows the multi-task processing model to learn the "knowledge" of each teacher model, ensuring that the model can be applied to multiple tasks without relying on fully labeled data.

[0049] Finally, in step S203, the multi-task processing model can be optimized and trained based on the aforementioned target loss. Specifically, the multi-task processing model can be trained using this target loss until it converges.

[0050] Therefore, the solution of this invention breaks through the conventional thinking of multi-task processing model training. It cleverly trains the multi-task processing model using readily available unlabeled and mixed-labeled data as training data and multiple teacher models supporting single-class text analysis tasks. Based on the trained multi-task processing model, it can process multiple types of text analysis tasks in speech synthesis. Thus, the solution of this application, through the training of a multi-task processing model, enables its simultaneous application to multiple tasks, breaking away from the traditional pipelined analysis approach. This not only saves resources such as text analysis time and space, but also makes the architecture of the multi-task processing model, which can simultaneously perform multi-task analysis, much simpler and easier to maintain than the pipelined system architecture. Furthermore, the multi-task processing model, which effectively saves resources and is easy to maintain, has greater universality, especially suitable for resource-constrained scenarios with high speech quality requirements, thus better achieving a balance between performance and quality.

[0051] Figure 3 A method 300 for training a multi-task processing model according to another embodiment of the present invention is illustrated schematically. It will be understood that method 300 is a... Figure 2 Further limitations and / or extensions of Chinese method 200. Therefore, the foregoing is combined with Figure 2 The relevant detailed descriptions also apply to the following text.

[0052] like Figure 3 As shown, in step S301, unlabeled data, mixed-label data, and multiple teacher models related to the multi-task processing model can be obtained. As mentioned above, the training data for the multi-task processing model can preferably be unlabeled data and mixed-label data. The unlabeled data can include some text corpora without task labels, while the mixed-label data is labeled with at least some task labels, such as only one task label, or two or more partial task labels. In some exemplary scenarios, the specific data sources for unlabeled data are relatively wide, and there are many ways to obtain it. For example, it can be obtained by performing simple corpus cleaning on some open-source text corpora. Mixed-label data is also relatively easy to obtain compared to complete labeled data. For example, data with partial task labels can be filtered from open-source text corpora. Of course, the mixed-label data can also be obtained through manual annotation. Since only a single or partial task label is annotated, the required mixed-label data can be obtained without excessive waste of resources. It should be noted that the description of the acquisition process of unlabeled data and mixed-label data here is only an illustrative example, and the solution of the present invention is not limited thereto.

[0053] Next, at step S302, in response to the training data containing the aforementioned unlabeled data, a first target loss can be determined based on the prediction results of each of the aforementioned teacher models and the multi-task processing model for the unlabeled data. In some embodiments, a first distillation loss can be constructed based on the prediction results of each teacher model and the multi-task processing model for the unlabeled data, and the aforementioned first target loss can be determined based on the first distillation loss to guide the multi-task processing model to learn each teacher model based on the first distillation loss.

[0054] For example, in some exemplary scenarios, multiple teacher models may include teacher models performing task A, task B, and task C, etc. Unlabeled data can be input into these three teacher models and a multi-task processing model acting as a student model for prediction. Assuming the prediction result of the teacher model for task A corresponds to the probability distribution P_teacher(A), and the prediction result of the multi-task student model for task A corresponds to the probability P_student(A), similarly, there are probability distributions P_teacher(B), P_teacher(C), P_student(B), and P_student(C) for tasks B and C, the first target loss can be determined based on the probability distribution results output by multiple teacher models and the multi-task processing model.

[0055] In some embodiments, a distillation loss can be constructed for each task, and then the distillation losses of all tasks can be weighted and summed to obtain the first objective loss mentioned above. For example, taking KL divergence as the distillation loss for any task, the distillation loss for that task can be obtained using the following formula:

[0056] In the formula, P corresponds to the probability distribution P_teacher(A), q corresponds to the distribution P_student(A), and x represents each unlabeled data sample. Similarly, tasks B and C will also construct their distillation losses based on this formula. Then, the distillation losses of the three tasks are weighted and summed to obtain the multi-task distillation loss (i.e., the first objective loss). It should be noted that the detailed description of the first objective loss here is only illustrative, and the solution of this invention is not limited to this. For example, the text analysis tasks involved are not limited to the three tasks in the embodiment, and the number of corresponding teacher models can also be adjusted according to the number of tasks. Furthermore, the distillation loss corresponding to each task is not limited to KL divergence and can be adjusted according to specific needs.

[0057] In step S303, in response to the training data containing mixed-label data, a second target loss can be determined based on the prediction results of the partial teacher model and the multi-task processing model for the mixed-label data. In some embodiments, firstly, the partially labeled data can be input into the aforementioned multi-task processing model and the teacher model used to perform the task corresponding to the missing label of the data for prediction, and a second distillation loss can be constructed based on the prediction results. Next, the cross-entropy loss between the prediction result of the multi-task processing model for the mixed-label data and the corresponding labeled task label can be calculated, and the aforementioned second target loss can be determined based on the second distillation loss and the cross-entropy loss.

[0058] For example, in some exemplary scenarios, suppose the mixed-label data includes data labeled only for task A, data labeled only for task B, and data labeled only for task C. For the data labeled only for task A, this data can be input into a multi-task processing model performing task B, a teacher model performing task C, and a student model for prediction. For tasks B and C, the second distillation loss for task B can be constructed using the prediction results of the teacher model performing task B and the multi-task processing model. Similarly, for task C, the second distillation loss for task C can be constructed using the prediction results of the teacher model performing task C and the multi-task processing model's prediction results for task B. Then, the cross-entropy loss is calculated using the prediction results of the multi-task processing model for the data labeled for task A and the labels for task A. Likewise, for the data labeled only for task B, the second distillation loss for task A and task C, as well as the cross-entropy loss for task B, can be constructed. For the data labeled only for task C, the second distillation loss for task A and task B, as well as the cross-entropy loss for task C, can be constructed. Finally, the second distillation loss and cross-entropy loss obtained from each task are weighted and summed to obtain the aforementioned second objective loss. It should be noted that the detailed description of the second objective loss here is merely illustrative, and the solution of this invention is not limited thereto. For example, the text analysis tasks involved are not limited to the three tasks in the embodiment, and the number of corresponding teacher models can be adjusted according to the number of tasks. Furthermore, the distillation loss corresponding to each task can be KL divergence or other types of loss functions.

[0059] Finally, in step S304, the multi-task processing model can be optimized and trained using the first target loss determined for unlabeled data and the second target loss determined for mixed-label data, based on the aforementioned teacher model and multi-task processing model. Specifically, in some embodiments, the parameters of the multi-task processing model can be iteratively optimized based on the aforementioned first and second target losses and combined with gradient backpropagation until the multi-task processing model converges, thereby completing the optimization and training of the multi-task processing model.

[0060] Therefore, in the training of the multi-task processing model, it does not rely on the difficult-to-obtain multi-task labeled data, but cleverly uses easily accessible unlabeled data and mixed labeled data combined with semi-supervised distillation technology to train the multi-task processing model. This not only solves the problem of the difficulty in obtaining multi-task data in the multi-task processing model, but also enables the multi-task processing model to effectively learn the capabilities of the teacher model based on distillation of a large amount of unlabeled data. At the same time, the integration of mixed labeled data learning also allows the multi-task processing model to learn in combination with actual labeled data during the distillation learning process, so that the final model will not have large deviations and is more stable.

[0061] Figure 4 A flowchart illustrating a method 400 for training a multi-task processing model according to another embodiment of the present invention is shown schematically. It is understood that method 400 is a further limitation or extension of methods 200 and 300. Therefore, the foregoing is combined with... Figure 2 and Figure 3 The relevant detailed descriptions also apply to the following text.

[0062] like Figure 4 As shown, in step S401, the resource configuration of the electronic device to which the multi-task processing model is applied can be obtained. Specifically, the resource configuration of the electronic device may include memory, processing performance (which can be determined based on processor type, etc.), or other basic configurations (such as the system architecture supported by the device, etc.).

[0063] Next, at step 402, the multi-task processing model can be optimized based on the aforementioned resource configuration. In some embodiments, the architecture of the teacher model, or the architecture of the multi-task processing model, or both the teacher model and the multi-task processing model can be adjusted based on the aforementioned resource configuration. For example, for devices with limited resources, a relatively simple basic neural network architecture can be used to construct the teacher model and / or the multi-task processing model. For devices with relatively higher resource configurations, a more complex basic neural network model can be used to construct the teacher model and / or the multi-task processing model.

[0064] In other embodiments, the number of tasks and / or task execution priorities for multiple text analysis tasks in speech synthesis can be adjusted based on resource configuration. For example, for devices with limited resources, individual tasks that consume fewer resources or are easier to analyze can be handled separately by the device, and then a multi-task processing model can be trained to handle the other multiple tasks, thereby reducing the number of tasks handled by the task processing model. Alternatively, when resources are limited, when training multiple text analysis tasks, priority can be assigned to the execution order of tasks based on their importance, so that resource utilization can be maximized in the subsequent inference stage. This ensures that the trained multi-task analysis model is highly adaptable to electronic devices.

[0065] Next, in step S403, the training data for the aforementioned multi-task processing model and multiple teacher models can be obtained. The multi-task processing model can be configured to perform multiple types of text analysis tasks in speech synthesis, while each teacher model can be configured to perform a single type of the aforementioned text analysis task. As mentioned earlier, the type and number of text analysis tasks are not specifically limited here; for example, they may include word segmentation analysis tasks, multi-speech analysis tasks, prosodic analysis tasks, or other text analysis tasks. The training data preferably includes unlabeled data and mixed-label data; for specific definitions or acquisition methods of unlabeled data and mixed-label data, please refer to the preceding text. Figure 2 or Figure 3 The relevant details will not be specified here.

[0066] Furthermore, the specific type of electronic device is not limited here; for example, it can include all devices that require text analysis and / or speech synthesis. That is, the multi-task processing model trained in this implementation has good versatility, especially adaptable to devices with limited resources and requiring high speech quality, such as dictionary pens.

[0067] Then, in step S404, the target loss for the multi-task processing model can be determined based on the prediction results of the aforementioned teacher model and multi-task processing model on the training data. As mentioned earlier, the target loss here involves a first target loss determined using the teacher model and multi-task processing model for unlabeled data and a second target loss determined for mixed-label data. For details on the determination process of the first and second target losses, please refer to [link to relevant documentation]. Figure 3 The detailed descriptions of steps S302 and S303 will not be repeated here.

[0068] Finally, in step S405, the multi-task processing model can be optimized and trained based on the aforementioned target loss. In some embodiments, the target loss involves the first target loss and the second target loss mentioned above. Therefore, the parameters of the multi-task processing model can be optimized iteratively using the first target loss and the second target loss in conjunction with gradient backpropagation, thereby achieving optimized training of the multi-task processing model.

[0069] Therefore, during the training of the multi-task processing model, the configuration information of the electronic device on which the model is applied can be used to further adjust the model, thereby ensuring that the multi-task processing model can be effectively implemented and applied on that electronic device. In particular, high-quality speech can still be obtained for some devices with limited resources, such as dictionary pens, achieving a balance between performance and quality.

[0070] Figure 5schematically shows a flowchart of a method 500 for training a multi-task processing model according to another embodiment of the present invention. It can be understood that the method 500 can be understood as a specific technical implementation of the method 200 or the method 300. Therefore, the above combination Figure 2 and Figure 3 the relevant detailed descriptions in also apply to the following.

[0071] As shown in Figure 5 , at step S501, the teacher model may be a plurality of BERT models performing single tasks respectively obtained by fine-tuning based on a small amount of labeled task data. Wherein, the language representation model (Bidirectional Encoder Representation from Transformers, BERT for short) is a large-scale pre-trained language model based on the transformer structure. By constructing a masked word prediction task, the model learns powerful semantic representation capability. Therefore, a very good effect can be obtained on downstream tasks by means of fine-tuning. In some exemplary scenarios, these BERT models that perform single tasks such as polyphonic character analysis, word segmentation analysis, prosodic analysis, etc. may adopt a multi-layer (for example, 12-layer, etc.) network architecture.

[0072] In some embodiments, labeled training data for each type of text analysis task may be obtained. And the basic language model is fine-tuned by using the labeled training data of each type of text analysis task, so as to obtain the corresponding teacher model that performs each type of text analysis task. Taking a BERT model performing single-task polyphonic character analysis (hereinafter referred to as the single-task polyphonic character model) as an example, the fine-tuning process of the BERT model performing single tasks is illustrated: a) First, the training data format is an input-output data pair (that is, labeled training data) composed of an input Chinese character sequence and an output label sequence. For example, the sentence "Field measurement requires acting according to one's capability." contains four polyphonic characters: two "量", one "要" and one "行", and the others are non-polyphonic characters. The corresponding output label sequence is (_, _, _, liang2, yao4, liang4, _, li4, _, xing2). Wherein, the label corresponding to non-polyphonic characters is represented by the symbol "_" by default.

[0073] (b) After constructing the training data for the aforementioned polyphonic characters, a single-task polyphonic character model with optimal performance is obtained by fine-tuning a BERT model (i.e., the base language model) with a small number of parameters. Preferably, this single-task polyphonic character model can be obtained by fine-tuning the BERT-base model. The BERT-base model is a type of BERT model, corresponding to a BERT model with 100M parameters. Of course, the description of the process for obtaining the single-task polyphonic character model here is merely illustrative, and the solution of this invention is not limited thereto. For example, other BERT models with small numbers of parameters (such as BERT-small, BERT-tiny, etc.) can also be selected and fine-tuned according to application requirements to obtain the single-task polyphonic character model.

[0074] Similarly, for single-task word segmentation and single-task prosody models, by constructing word segmentation and prosody training data respectively, and then fine-tuning the BERT-base model, the best-performing single-task polyphonic character and single-task prosody models are obtained respectively, which can be used as the teacher model for distillation.

[0075] Next, in step S502, BERT can be distilled into a student model (i.e., a multi-task processing model) based on a large amount of unlabeled data and mixed labeled data through a multi-task semi-supervised hybrid process. In some embodiments, the student model can be the albbert_tiny_zh model. The albbert_tiny_zh model is a type of BERT model, and using the albbert_tiny_zh model as the student model results in a small number of parameters required, making it more suitable for embedded resource-constrained scenarios.

[0076] In practical applications, after obtaining three optimal single-task models for polyphonic characters, word segmentation, and prosody, these three models can be used as teacher models. Furthermore, through multi-task semi-supervised distillation using unlabeled and mixed-labeled data, the model is converted into the student model abert_tiny_zh, as detailed below: a) First, unlabeled data refers to text data without labels. There is a lot of this type of data, and it is very easy to obtain. It can be obtained through some open-source text corpora and simple corpus cleaning steps.

[0077] b) Mixed label data refers to data for three types of tasks, where each task has its own (input, output) data pairs. For example, the data pair for the polyphonic character task is in the following form (input: Field measurement should be done according to one's capacity, output: _, _, _,liang2, yao4, liang4, _, li4, _, xing2). The data pair for the word segmentation task is in the following form (input: Study the origin of life, output: B M B M SB M). The data pair for the prosody task is in the following form (input: The poem describes the quiet scene in the suburbs of Chuzhou in spring, output: sp0 sp1 sp0 sp0 sp3 sp0 sp1 sp0 sp1 sp0 sp1 sp2sp0 sp1 sp0 sp4). The specific forms of labels included in the data of mixed label data and the description of the number of tasks herein are merely exemplary illustrations, and the solution of the present invention is not limited thereto.

[0078] c) Since both traditional multi-task models and distillation methods need to rely on a large amount of multi-task label data, and such data is difficult to obtain. Therefore, it is necessary to break this technical inertial thinking. A large amount of unlabeled data and easily obtainable mixed label data can be obtained herein, and combined with the multi-task semi-supervised distillation method, a high-performance multi-task model with a small number of parameters can finally be obtained based on the existing data resources.

[0079] In some embodiments, the specific multi-task semi-supervised distillation method comprises: for unlabeled data, the data can be simultaneously input into three single-task teacher models and a multi-task student model to obtain corresponding teacher prediction results of three tasks and student prediction results; a distillation loss (i.e., distillation loss) is constructed based on the teacher prediction results and the student prediction results, so that the student model can learn information from the teacher models. For mixed labeled data of three tasks, for example, labeled data of polyphonic characters, the data is respectively input into a single-task word segmentation teacher model, a single-task prosody model and a multi-task student model. A distillation loss for the polyphonic character task is constructed based on a prediction result of a teacher model (from a single-task polyphonic character teacher model) and a prediction result of the student model for the polyphonic character task; meanwhile, a cross-entropy loss (i.e., cross-entropy loss) is constructed based on the polyphonic character prediction result of the student model and the polyphonic character labeled data, so as to further use the polyphonic character labeled data to guide model learning. Similarly, for labeled word segmentation data and labeled prosody data, distillation losses and cross-entropy losses for the word segmentation task and the prosody task are constructed simultaneously to guide the student model to transfer information from the teacher models and learn with the labeled data. Based on this method, a large amount of unlabeled data and easily obtainable mixed labeled data are effectively utilized, which solves the problem that multi-task data is difficult to obtain in multi-task models. Meanwhile, through distillation based on a large amount of data, the student model effectively learns the capabilities of the teacher models; and incorporating learning with mixed labeled data also enables the student model to combine actual labeled data for learning during the distillation learning process. Eventually, the student model will not have large deviations and is more stable, and finally a model that can perform multi-task prediction simultaneously, has relatively fewer parameters, and achieves relatively better performance is obtained.

[0080] In step S503, the student model alber_tiny_zh may be fine-tuned based on labeled multi-task data. After obtaining the multi-task albert_tiny_zh model through step S502, a small amount of manually annotated multi-task data may be used to further fine-tune the model, so as to further improve the consistency and stability of the multi-task model. Wherein, each piece of data in the multi-task data can be understood as a piece of text data simultaneously annotated with multiple task label data. For example, the corpus "野外测量要量力而行" (Field measurement should be done according to one's capability) needs to be simultaneously annotated with a polyphonic character label sequence (_, _, _, liang2, yao4, liang4, _, li4, _, xing2), a word segmentation label sequence (B M B M S B M B M), and a prosody label sequence (sp0 sp1 sp0 sp2 sp0 sp0 sp1 sp0 sp3). This is merely an illustrative description of the data format in the multi-task data, and no limitation is imposed on the specific number of tasks and the specific label format.

[0081] By constructing a multi-task processing mode based on the BERT model, the traditional pipeline combination approach is broken, allowing the multi-task processing model to be applied to multiple tasks simultaneously. This not only saves time and space but also simplifies the system structure and makes it easier to maintain. Furthermore, a multi-task semi-supervised distillation technique is proposed, further improving the performance of models with the same number of parameters.

[0082] Figure 6 A flowchart illustrating a method 600 for speech synthesis based on a multi-task processing model according to an embodiment of the present invention is shown schematically. It is understood that the multi-task processing model in method 600 can be trained using method 200, method 300, method 400, or method 500. Therefore, the preceding text, in conjunction with... Figures 2-5 The relevant details described in the text also apply to the following text.

[0083] like Figure 6 As shown, in step S601, the trained multi-task processing model can be obtained. It can be understood that this multi-task processing model can be obtained through... Figures 2-5 The training was performed using any of the methods shown. For details of the training process, please refer to the relevant descriptions above. It will not be repeated here.

[0084] Next, in step S602, a multi-task processing model can be used to perform multi-task text analysis on the acquired text to obtain prediction results. In step S603, the speech synthesis result of the text can be determined based on the prediction results of the multi-task processing model. After obtaining the prediction results for the multi-task tasks, traditional speech synthesis techniques can be used to obtain the final speech synthesis result based on the prediction results. This multi-task processing model can simultaneously perform multiple text analysis tasks. Using this multi-task processing model for text analysis processing can effectively reduce the consumption of system memory and significantly reduce inference time, while ensuring that the multi-task effect and single-task effect can reach a comparable level. Therefore, a speech synthesis system can be used to synthesize high-quality speech with low pronunciation errors, natural rhythm, and rich emotion in devices with limited resource configurations (such as offline devices).

[0085] Exemplary device After introducing the method of exemplary embodiments of the present invention, the following references are made. Figure 7 The present invention describes related products for postgraduate entrance examination preparation management based on exemplary embodiments of the present invention.

[0086] Figure 7 A schematic block diagram of an electronic device 700 according to an embodiment of the present invention is shown. Figure 7As shown, the electronic device 700 may include a processor 701 and a memory 702. The memory 702 stores computer instructions for training a multi-task processing model and / or speech synthesis. When these computer instructions are executed by the processor 701, they cause the electronic device 700 to perform actions according to the foregoing description. Figures 2 to 5 and / or Figure 6 The described method. For example, in some embodiments, the electronic device 700 can acquire training data about a multi-task processing model and multiple teacher models, calculate the target loss, optimize the training of the multi-task processing model, and / or perform speech synthesis based on the multi-task processing model, etc. Based on this, the electronic device 700 can support the training of the multi-task processing model and / or speech synthesis using only the multi-task processing model.

[0087] In some implementation scenarios, electronic device 700 may include resource-constrained electronic devices (such as dictionary pens) to utilize multi-task processing models for speech synthesis, thereby achieving high-quality speech synthesis based on limited resources. Of course, the present invention does not limit the structural design of electronic device 700; for example, it can also be a resource-unconstrained device that can simultaneously support the training and inference of multi-task processing models.

[0088] It should be noted that although several devices or sub-devices of the device have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the invention, the features and functions of two or more devices described above can be embodied in one device. Conversely, the features and functions of one device described above can be further divided and embodied by multiple devices.

[0089] The use of the verbs "including" and "contains" and their inflections in the application documents does not preclude the existence of elements or steps other than those described in the application documents. The article "a" or "one" preceding an element does not preclude the existence of multiple such elements.

[0090] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be interpreted in the broadest sense, thereby encompassing all such modifications and equivalent structures and functions.

Claims

1. A method for training a multi-task processing model, characterized in that, include: Acquire training data for the multi-task processing model and multiple teacher models, wherein the multi-task processing model is configured to perform multiple text analysis tasks in speech synthesis, and each teacher model is configured to perform a single type of text analysis task; Based on the prediction results of the teacher model and the multi-task processing model for the training data, a target loss for the multi-task processing model is determined. The training data includes unlabeled data and mixed-label data, where the mixed-label data contains data labeled with at least some task labels. Determining the target loss for the multi-task processing model includes: in response to the training data containing the unlabeled data, determining a first target loss based on the prediction results of each of the teacher model and the multi-task processing model for the unlabeled data; and in response to the training data containing the mixed-label data, determining a second target loss based on the prediction results of a portion of the teacher model and the multi-task processing model for the mixed-label data; and The multi-task processing model is optimized and trained using the teacher model and the multi-task processing model, based on the first target loss determined for the unlabeled data and the second target loss determined for the mixed-label data.

2. The method according to claim 1, characterized in that, The first target loss is determined based on the prediction results of each of the teacher models and the multi-task processing models for the unlabeled data, including: A first distillation loss is constructed based on the prediction results of each teacher model and the multi-task processing model on the unlabeled data, and a first target loss is determined based on the first distillation loss, so as to guide the multi-task processing model to learn each teacher model based on the first distillation loss.

3. The method according to claim 1, characterized in that, The second objective loss is determined based on the prediction results of the teacher model and the multi-task processing model on the mixed-label data, as well as the labeled task labels, including: Data with partial labels are input into the multi-task processing model and the teacher model used to perform the task corresponding to the missing data labels for prediction, and a second distillation loss is constructed based on the prediction results; and Calculate the cross-entropy loss between the prediction result of the multi-task processing model for the mixed-label data and the corresponding labeled task labels, and determine the second target loss based on the second distillation loss and the cross-entropy loss.

4. The method according to claim 1, characterized in that, Obtaining multiple teacher models for the multi-task processing model includes: Obtain labeled training data for each type of text analysis task; and The base language model is fine-tuned using labeled training data for each type of text analysis task to obtain the corresponding teacher model for performing each type of text analysis task.

5. The method according to claim 4, characterized in that, The multi-type text analysis tasks include polyphonic character analysis, word segmentation analysis, and prosodic analysis, and the basic language model includes the BERT-Base model.

6. The method according to claim 1, characterized in that, The training data also includes multi-task data, wherein the data in the multi-task data is labeled with the labels of all tasks, and the method further includes: The multi-task processing model is fine-tuned using the multi-task data.

7. The method according to claim 1, characterized in that, The multi-task processing model is applied to an electronic device, and the method further includes: Obtain the resource configuration of the electronic device; and The multi-task processing model is optimized based on the resource configuration.

8. The method according to claim 7, characterized in that, Optimizing the multi-task processing model based on the resource allocation includes: Adjust the model architecture of the teacher model and / or multi-task processing model according to the resource configuration; and / or Adjust the number of tasks and / or the execution priority of the various text analysis tasks according to the resource configuration.

9. A method for speech synthesis, characterized in that, include: Obtain a trained multi-task processing model, wherein the multi-task processing model is trained by the method of any one of claims 1 to 8; The acquired text is processed using the aforementioned multi-task processing model to obtain prediction results; and The speech synthesis result of the text is determined based on the prediction results of the multi-task processing model.

10. An electronic device, characterized in that, include: processor; as well as A memory storing computer instructions for training a multitasking model and / or speech synthesis, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1-8 and / or the method according to claim 9.

11. A computer-readable storage medium, characterized in that, The method includes program instructions for training a multi-task processing model and / or speech synthesis, which, when executed by a processor, cause the implementation of the method according to any one of claims 1-8 and / or the method according to claim 9.

Images