A training method, device, electronic device and storage medium for a speech synthesis model
By local training of the speech synthesis model, the problem that the speech text cannot end correctly is solved, the automatic update of the speech synthesis model is realized, and the stability and efficiency of speech synthesis are improved.
Patent Information
- Application Number
- CN202010835266.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-19
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-08-19
AI Technical Summary
The speech synthesis model may not end correctly when processing speech text, resulting in a large number of invalid speeches being generated in subsequent stages, and the prior art is difficult to effectively solve this problem.
The speech text is processed through the speech synthesis model to determine whether the speech statement ends correctly. If it is incorrect, determine the correct ending position, store it in the exception training library, and locally train the model based on the library to update the model to improve the correctness The probability of the ending.
Automatic update of the speech synthesis model is realized, which increases the probability of correct ending, improves the stability of speech synthesis, and reduces maintenance costs and data costs.
Smart Images

Figure CN111968616B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech synthesis technology, and in particular to a training method, device, electronic device and storage medium for a speech synthesis model. Background Art
[0002] The speech synthesis model is a neural network model that can convert speech sentences into corresponding speech. When the speech text is processed by the speech synthesis model, it is possible that the speech sentence is not correctly ended, resulting in a large amount of invalid speech in the subsequent stage. Summary of the invention
[0003] To solve the above problems, the present application proposes a training method, device, electronic device and storage medium for a speech synthesis model.
[0004] In a first aspect, an embodiment of the present application provides a method for training a speech synthesis model, comprising:
[0005] Processing the speech text through a speech synthesis model to obtain speech corresponding to a speech sentence in the speech text;
[0006] Based on the length of the speech, determining whether the speech sentence ends correctly;
[0007] If it is determined that the voice sentence does not end correctly, determining the correct end position of the voice sentence;
[0008] The speech sentence and the correct end position of the speech sentence are stored in an abnormal training library, and the speech synthesis model is locally trained based on the abnormal training library to obtain an updated speech synthesis model.
[0009] In a second aspect, an embodiment of the present application provides a training device for a speech synthesis model, comprising:
[0010] A processing unit, used to process the speech text through a speech synthesis model to obtain speech corresponding to a speech sentence in the speech text;
[0011] A judgment unit, used for judging whether the speech sentence ends correctly based on the length of the speech;
[0012] A determination unit, configured to determine a correct end position of the speech sentence if it is determined that the speech sentence does not end correctly;
[0013] The training unit is used to store the speech sentence and the correct end position of the speech sentence in an abnormal training library, and perform local training on the speech synthesis model based on the abnormal training library to obtain an updated speech synthesis model.
[0014] In a third aspect, an embodiment of the present application provides an electronic device, including:
[0015] one or more processors;
[0016] a memory communicatively coupled to the one or more processors;
[0017] One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-mentioned method.
[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the above-described method when executed by a processor.
[0019] By adopting the above technical solution of the embodiment of the present application, when the speech text is processed by the speech synthesis model, it is determined whether the speech sentence ends correctly; for the speech sentence that does not end correctly, the correct end position of the speech sentence is determined, and the speech synthesis model is locally trained based on the correct end position of the speech sentence, thereby realizing automatic updating of the speech synthesis model. The updated speech synthesis model can greatly increase the probability of correct ending, thereby improving the stability of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 Schematic diagram of the implementation process of the training method of the speech synthesis model provided in the embodiment of the present application Figure 1 ;
[0021] Figure 2 Schematic diagram of the implementation process of the training method of the speech synthesis model provided in the embodiment of the present application Figure 2 ;
[0022] Figure 3 A schematic diagram of the structure of a training device for a speech synthesis model provided in an embodiment of the present application;
[0023] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0025] In some processes described in the specification and claims of this application and the above-mentioned drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these processes may include more or fewer operations, and these operations may be performed sequentially or in parallel.
[0026] The speech synthesis model (i.e., speech synthesis system) is implemented based on an end-to-end neural network. Therefore, the speech synthesis model can also be called a speech synthesis neural network model. On the one hand, the current training of the speech synthesis model requires a large amount of high-quality training speech (e.g., more than 10 hours of training speech) as training samples, which makes the training cost very high. On the other hand, once a problem occurs in the speech synthesis model, it is difficult to solve it. For example, an important problem that is often encountered is that after the speech synthesis of the speech text is completed, part of the generated speech does not end correctly, resulting in a large amount of invalid speech in the subsequent stage.
[0027] Based on this, the embodiments of the present application provide a training method, device, electronic device and storage medium for a speech synthesis model, which updates the speech synthesis model through a fully automatic update method. The updated speech synthesis model can solve the problem of being unable to end the speech synthesis.
[0028] Figure 1 Schematic diagram of the implementation process of the training method of the speech synthesis model provided in the embodiment of the present application Figure 1 ,like Figure 1 As shown, the method includes:
[0029] Step 101: Process the speech text through a speech synthesis model to obtain the speech corresponding to a speech sentence in the speech text.
[0030] In the embodiment of the present application, the speech synthesis model can realize speech synthesis, which is also called text-to-speech (TTS), and can convert any input speech text into corresponding speech.
[0031] In an optional manner of the present application, when the speech synthesis model receives a speech synthesis request message, it starts to process the speech text. Here, the speech synthesis request message can be triggered by a user.
[0032] In an embodiment of the present application, when the speech text is processed by the speech synthesis model, whether the end is required is determined according to the end judgment symbol (or called the end marker) of each speech sentence in the speech text.
[0033] It should be noted that the speech synthesis model may give a correct ending to a certain speech sentence, or it may give an incorrect ending to a certain speech sentence. It is not desirable to give a speech sentence with an incorrect ending. Therefore, it is necessary to re-update the speech synthesis model to improve the accuracy of the speech synthesis model in correctly ending the speech sentence. The technical solution of the embodiment of the present application updates the speech synthesis model according to the following steps 102 to 104.
[0034] Step 102: Based on the length of the speech, determine whether the speech sentence ends correctly.
[0035] In the embodiment of the present application, whether the voice sentence ends correctly is determined by the following method:
[0036] a) determining whether the length of the speech is greater than or equal to a first target threshold; b1) if the length of the speech is greater than or equal to the first target threshold, determining that the speech sentence does not end correctly; b2) if the length of the speech is less than the first target threshold, determining that the speech sentence ends correctly.
[0037] Here, a threshold (i.e., the first target threshold) is preset, and the unit of the threshold is duration. It is determined whether the length of the synthesized speech reaches (or is greater than or equal to) the threshold. If it reaches, it is determined that the speech synthesis model does not perform a correct ending operation on the speech sentence. If it does not reach, it is determined that the speech synthesis model performs a correct ending operation on the speech sentence, and the synthesis processing of the next speech sentence can be continued.
[0038] Step 103: If it is determined that the voice sentence does not end correctly, then determine the correct end position of the voice sentence.
[0039] In an embodiment of the present application, the correct end position of the voice sentence is determined by: recognizing the voice; comparing the recognition result of the voice with the voice text containing the voice sentence, and determining the correct end position of the voice sentence based on the comparison result.
[0040] Here, the speech is recognized, and the recognition result obtained is the speech sentence corresponding to the speech. The speech sentence corresponding to the speech is compared with the speech text to detect the correct end position of the speech sentence.
[0041] For example, the recognition result of speech is: valid phoneme + invalid phoneme. The recognition result is compared with the speech text to determine that the correct end position is after the valid phoneme.
[0042] Step 104: storing the speech sentence and the correct end position of the speech sentence in an abnormal training library, and performing local training on the speech synthesis model based on the abnormal training library to obtain an updated speech synthesis model.
[0043] In the embodiment of the present application, by repeatedly executing the above steps 101 to 103, multiple voice sentences and their corresponding correct end positions can be obtained, and multiple voice sentences and their corresponding correct end positions can be stored in the abnormal training library.
[0044] In an optional manner, it is determined whether the number of speech sentences in the abnormal training library reaches a second target threshold; if the number of speech sentences in the abnormal training library reaches the second target threshold, the speech synthesis model is locally trained based on the abnormal training library.
[0045] In another optional manner, it is determined whether the time from the target time to the current time reaches a specified time; if the time from the target time to the current time reaches a specified time, the speech synthesis model is locally trained based on the abnormal training library. Here, the target time is the time when the speech synthesis model processes the first speech sentence in the text; or, the target time is the time when the abnormal training library is first updated.
[0046] In an embodiment of the present application, the local training of the speech synthesis model based on the abnormal training library includes: adding the training data in the abnormal training library to the original training data, and locally training the speech synthesis model. In an optional method, the speech synthesis model is locally trained using a migration technique (also referred to as local retraining). In this way, the problem that the speech sentence cannot end normally is solved without destroying the original normal synthesized speech effect of the speech synthesis model.
[0047] In the embodiment of the present application, after local training of the speech synthesis model is completed, an updated speech synthesis model is obtained, and the updated speech synthesis model replaces the previous old speech synthesis model to achieve automatic optimization of the speech synthesis model.
[0048] In the technical solution of the embodiment of the present application, the problem that the speech synthesis model cannot correctly end the language sentence is solved by the automatically updated speech synthesis model. The automatic update of the speech synthesis model does not require manual intervention, which reduces maintenance costs. In addition, there is no need to expand the original training data, which greatly reduces data costs. Furthermore, users can perform local training on the speech synthesis model according to their own strategies, thereby continuously improving the stability of speech synthesis and greatly improving the product experience.
[0049] Figure 2Schematic diagram of the implementation process of the training method of the speech synthesis model provided in the embodiment of the present application Figure 2 ,like Figure 2 As shown, the method includes:
[0050] Step 201: Process the speech text through a speech synthesis model to obtain the speech corresponding to a speech sentence in the speech text.
[0051] In the embodiment of the present application, the speech synthesis model can realize speech synthesis, which is also called TTS, and can convert any input speech text into corresponding speech.
[0052] In an optional manner of the present application, when the speech synthesis model receives a speech synthesis request message, it starts to process the speech text. Here, the speech synthesis request message can be triggered by a user.
[0053] In an embodiment of the present application, when the speech text is processed by the speech synthesis model, whether the end is required is determined according to the end judgment symbol (or called the end marker) of each speech sentence in the speech text.
[0054] It should be noted that the speech synthesis model may give a correct ending to a certain speech sentence, or it may give an incorrect ending to a certain speech sentence. It is not desirable to give a speech sentence with an incorrect ending, so it is necessary to re-update the speech synthesis model to improve the accuracy of the speech synthesis model in giving a correct ending to the speech sentence. The technical solution of the embodiment of the present application updates the speech synthesis model according to the following steps.
[0055] Step 202: Determine whether the length of the speech reaches a first target threshold, if so, execute step 203, if not, execute step 201.
[0056] Here, a threshold (i.e., the first target threshold) is preset, and the unit of the threshold is duration. It is determined whether the length of the synthesized speech reaches (or is greater than or equal to) the threshold. If it reaches, it is determined that the speech synthesis model does not perform a correct ending operation on the speech sentence. If it does not reach, it is determined that the speech synthesis model performs a correct ending operation on the speech sentence, and the synthesis processing of the next speech sentence can be continued.
[0057] Step 203: Recognize the speech; compare the recognition result of the speech with the speech text containing the speech sentence, and determine the correct end position of the speech sentence based on the comparison result.
[0058] Step 204: Store the voice sentence and the correct end position of the voice sentence in an abnormal training library.
[0059] Step 205: Determine whether the number of speech sentences in the abnormal training library reaches the second target threshold; if so, execute step 206; if not, execute step 201.
[0060] Step 206: Perform local training on the speech synthesis model based on the abnormal training library to obtain an updated speech synthesis model.
[0061] In an embodiment of the present application, the local training of the speech synthesis model based on the abnormal training library includes: adding the training data in the abnormal training library to the original training data to perform local training on the speech synthesis model. In an optional manner, the speech synthesis model is locally trained using a migration technique. In this way, the problem that the speech sentence cannot end normally is solved without destroying the original normal synthesized speech effect of the speech synthesis model.
[0062] In the embodiment of the present application, after local training of the speech synthesis model is completed, an updated speech synthesis model is obtained, and the updated speech synthesis model replaces the previous old speech synthesis model to achieve automatic optimization of the speech synthesis model.
[0063] Figure 3 A schematic diagram of the structure of the training device for the speech synthesis model provided in the embodiment of the present application is shown in FIG. Figure 3 As shown, the device comprises:
[0064] The processing unit 301 is used to process the speech text through a speech synthesis model to obtain a speech corresponding to a speech sentence in the speech text;
[0065] A judging unit 302, configured to judge whether the speech sentence ends correctly based on the length of the speech;
[0066] The determining unit 303 is used to determine the correct end position of the speech sentence if it is determined that the speech sentence does not end correctly;
[0067] The training unit 304 is used to store the speech sentence and the correct end position of the speech sentence in an abnormal training library, and perform local training on the speech synthesis model based on the abnormal training library to obtain an updated speech synthesis model.
[0068] In an optional manner, the judgment unit 302 is specifically used to judge whether the length of the speech is greater than or equal to a first target threshold; if the length of the speech is greater than or equal to the first target threshold, it is determined that the speech sentence does not end correctly; if the length of the speech is less than the first target threshold, it is determined that the speech sentence ends correctly.
[0069] In an optional manner, the determination unit 303 is specifically configured to recognize the speech; compare the recognition result of the speech with the speech text containing the speech sentence, and determine the correct end position of the speech sentence based on the comparison result.
[0070] In an optional manner, the judging unit 302 is further configured to judge whether the number of speech sentences in the abnormal training library reaches a second target threshold;
[0071] The training unit 304 is configured to perform local training on the speech synthesis model based on the abnormal training library if the number of speech sentences in the abnormal training library reaches a second target threshold.
[0072] In an optional manner, the judging unit 302 is further configured to judge whether the time between the target time and the current time reaches a specified time;
[0073] The training unit 304 is used to perform local training on the speech synthesis model based on the abnormal training library if the duration between the target time and the current time reaches a specified duration.
[0074] In an optional manner, the target time is the time when the speech synthesis model processes the first speech sentence in the text; or, the target time is the time when the abnormal training library is first updated.
[0075] It should be noted here that the description of the above device embodiment items is similar to the above method description and has the same beneficial effects as the method embodiment, so it is not repeated here. For technical details not disclosed in the device embodiment of the present application, those skilled in the art should refer to the description of the method embodiment of the present application for understanding, and in order to save space, it is not repeated here.
[0076] An embodiment of the present application also provides an electronic device, comprising: one or more processors; a memory communicatively connected to the one or more processors; one or more applications; wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the above-described method.
[0077] In a specific example, the electronic device described in the embodiment of the present application may be specifically as follows: Figure 4In the structure shown, the electronic device at least includes a processor 41, a storage medium 42 and at least one external communication interface 43; the processor 41, the storage medium 42 and the external communication interface 43 are all connected via a bus 44. The processor 41 can be an electronic component with processing functions such as a microprocessor, a central processing unit, a digital signal processor or a programmable logic array. The storage medium stores a computer executable code, and the computer executable code can execute the method described in any of the above embodiments. In practical applications, the processing unit in the above scheme can be implemented by the processor 41.
[0078] It should be noted here that the description of the above electronic device embodiment is similar to the above method description and has the same beneficial effects as the method embodiment, so it is not repeated here. For technical details not disclosed in the electronic device embodiment of this application, those skilled in the art should refer to the description of the method embodiment of this application for understanding, and in order to save space, it is not repeated here.
[0079] The embodiment of the present application further provides a computer-readable storage medium storing a computer program, which implements the above-described method when executed by a processor.
[0080] Here, a computer-readable storage medium may be any device that can contain, store, communicate, propagate or transmit a program for use with or in conjunction with an instruction execution system, device or apparatus. More specific examples (a non-exhaustive list) of computer-readable storage media include the following: an electrical connection with one or more wirings (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), a fiber optic device, and a portable read-only memory (CDROM). In addition, a computer-readable storage medium may even be paper or other suitable medium on which the program may be printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting or, if necessary, processing in another suitable manner, and then stored in a computer memory.
[0081] It should be understood that a person skilled in the art can understand that all or part of the steps carried out in the method of the above-mentioned embodiment can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one of the steps of the method embodiment or a combination thereof.
[0082] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.
[0083] The embodiments described above are only a part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.
Claims
1. A method for training a speech synthesis model, characterized in that: The method comprises: Processing the speech text through a speech synthesis model to obtain speech corresponding to a speech sentence in the speech text; Based on the length of the speech, determining whether the speech sentence ends correctly; If it is determined that the voice sentence does not end correctly, determining the correct end position of the voice sentence; The speech sentence and the correct ending position of the speech sentence are stored in an abnormal training library, and the speech synthesis model is locally trained based on the abnormal training library to realize automatic updating of the speech synthesis model, improve the probability of the correct ending, and obtain an updated speech synthesis model; wherein the abnormal training library stores multiple speech sentences and their corresponding correct ending positions; Wherein, determining the correct end position of the voice sentence includes: Recognize the speech; compare the recognition result of the speech with the speech text where the speech sentence is located, and determine the correct end position of the speech sentence based on the comparison result.
2. The method according to claim 1, characterized in that: The step of determining whether the speech sentence ends correctly based on the length of the speech includes: Determining whether the length of the speech is greater than or equal to a first target threshold; If the length of the speech is greater than or equal to the first target threshold, it is determined that the speech sentence does not end correctly; If the length of the speech is less than the first target threshold, it is determined that the speech sentence ends correctly.
3. The method according to claim 1 or 2, characterized in that: The locally training the speech synthesis model based on the abnormal training library includes: Determining whether the number of speech sentences in the abnormal training library reaches a second target threshold; If the number of speech sentences in the abnormal training library reaches a second target threshold, the speech synthesis model is locally trained based on the abnormal training library.
4. The method according to claim 1 or 2, characterized in that: The locally training the speech synthesis model based on the abnormal training library includes: Determine whether the distance between the target time and the current time reaches the specified time; If the distance between the target time and the current time reaches a specified time, the speech synthesis model is locally trained based on the abnormal training library.
5. The method according to claim 4, characterized in that The target time is the time taken by the speech synthesis model to process the first speech sentence in the text; or The target time is the time when the abnormal training library is first updated.
6. A training device for a speech synthesis model, characterized in that: The device comprises: A processing unit, used to process the speech text through a speech synthesis model to obtain speech corresponding to a speech sentence in the speech text; A judgment unit, used for judging whether the speech sentence ends correctly based on the length of the speech; A determination unit, configured to determine a correct end position of the speech sentence if it is determined that the speech sentence does not end correctly; A training unit, used for storing the speech sentence and the correct end position of the speech sentence in an abnormal training library, and performing local training on the speech synthesis model based on the abnormal training library to realize automatic updating of the speech synthesis model, improve the probability of the correct ending, and obtain an updated speech synthesis model; wherein the abnormal training library stores a plurality of speech sentences and their corresponding correct end positions; The determining unit is specifically used to recognize the speech; compare the recognition result of the speech with the speech text where the speech sentence is located, and determine the correct end position of the speech sentence based on the comparison result.
7. The device according to claim 6, characterized in that The judgment unit is specifically used to judge whether the length of the speech is greater than or equal to a first target threshold; if the length of the speech is greater than or equal to the first target threshold, it is judged that the speech sentence does not end correctly; if the length of the speech is less than the first target threshold, it is judged that the speech sentence ends correctly.
8. The device according to claim 6 or 7, characterized in that The judging unit is further configured to judge whether the number of speech sentences in the abnormal training library reaches a second target threshold; The training unit is used to perform local training on the speech synthesis model based on the abnormal training library if the number of speech sentences in the abnormal training library reaches a second target threshold.
9. The device according to claim 6 or 7, characterized in that The judging unit is further used to judge whether the time between the target time and the current time reaches a specified time; The training unit is used to perform local training on the speech synthesis model based on the abnormal training library if the duration from the target time to the current time reaches a specified duration.
10. The device according to claim 9, characterized in that The target time is the time taken by the speech synthesis model to process the first speech sentence in the text; or The target time is the time when the abnormal training library is first updated.
11. An electronic device, characterized in that: include: one or more processors; a memory communicatively coupled to the one or more processors; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 5.
12. A computer-readable storage medium storing a computer program, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.
