Speech synthesis and its model training methods, devices, equipment and storage media

Abstract

This application discloses a speech synthesis method, apparatus, device, and storage medium for training its model, comprising: converting punctuation marks in input text into corresponding preset punctuation marks to obtain text samples; inputting the text samples into an initial speech synthesis model, processing the text samples through a text encoder of the initial speech synthesis model, and outputting text latent vectors, wherein the text latent vectors include punctuation latent vectors corresponding to the punctuation marks in the text samples, and the initial speech synthesis model does not include an explicit prosody prediction module and a related explicit prosody label module; and training the initial speech synthesis model based on the text latent vectors and preset speech prosody mapping relationships to obtain a target speech synthesis model.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a speech synthesis method, apparatus, device and storage medium for training its model. Background Technology

[0002] With the increasing prevalence of in-vehicle voice systems, the demand for speech synthesis functions in low-end vehicles is growing. However, due to cost constraints, the on-board devices in low-end vehicles suffer from low computing power and limited memory resources. This leads to frequent access to off-chip memory during model deployment, increasing latency and energy consumption, and potentially causing issues such as lag and inability to start normally. Existing VITS (Variational Inference with Adversarial Learning for End-to-End Text-to-Speech) models are difficult to adapt to the low-computing-power deployment requirements of low-end vehicles, limiting their widespread application in low-end in-vehicle scenarios.

[0003] Currently, lightweight improvements to the VITS model mainly focus on weight pruning, quantization, or inference engine optimization. These improvements cannot fundamentally solve the problem of excessive computing power and memory consumption caused by the prosody prediction module, making it difficult to meet the extreme lightweight requirements of low-end vehicle deployment.

[0004] Therefore, how to reduce the resource consumption of speech synthesis models is an urgent problem to be solved.

[0005] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0006] The main purpose of this application is to provide a speech synthesis method, apparatus, device and storage medium for training its model, aiming to solve the technical problem of how to reduce the resource consumption of speech synthesis models.

[0007] To achieve the above objectives, this application proposes a speech synthesis model training method, which includes: The punctuation marks in the input text are converted into corresponding preset punctuation marks to obtain text samples; The text sample is input into the initial speech synthesis model, and the text encoder of the initial speech synthesis model processes the text sample to output a text latent vector. The text latent vector includes the punctuation latent vectors corresponding to the punctuation marks in the text sample. The initial speech synthesis model does not include an explicit prosody prediction module and a related explicit prosody label module. The initial speech synthesis model is trained based on the text latent vectors and the preset speech prosody mapping relationship to obtain the target speech synthesis model.

[0008] In one embodiment, the text latent vector includes the text latent vectors corresponding to the characters in the text sample, and the step of processing the text sample by the text encoder of the initial speech synthesis model to output the text latent vector includes: The text sample is processed by the text encoder to obtain the text latent vector; The text encoder increases the weight of the encoded features corresponding to punctuation marks in the text sample and captures the position and type information of the punctuation marks in the text sample. The text encoder determines the punctuation latent vector based on the location and type information.

[0009] In one embodiment, the preset speech prosody mapping relationship includes the speech prosody association relationship between each preset punctuation mark and speech prosody features.

[0010] In one embodiment, the step of training the initial speech synthesis model based on the text latent vectors and a preset speech prosody mapping relationship to obtain the target speech synthesis model includes: The initial speech synthesis model is trained based on the text latent vectors and the preset speech prosody mapping relationship. The phoneme duration distribution in the random duration predictor of the initial speech synthesis model is dynamically adjusted based on the punctuation marks in the text samples to obtain the target speech synthesis model.

[0011] In one embodiment, the step of converting the punctuation marks in the input text into corresponding preset punctuation marks to obtain a text sample includes: Input the input text into the initial speech synthesis model; The input text is preprocessed by the punctuation integration module of the initial speech synthesis model to convert the punctuation marks in the text into corresponding preset punctuation marks, thereby obtaining a text sample.

[0012] In one embodiment, the step of converting the punctuation marks in the input text into corresponding preset punctuation marks to obtain a text sample includes: Obtain each punctuation mark in the input text; Based on the preset conversion relationship, each punctuation mark in the input text is converted into a preset punctuation mark of the corresponding type to obtain a text sample.

[0013] Furthermore, to achieve the above objectives, this application also proposes a speech synthesis method applied to a vehicle, wherein the vehicle is equipped with the aforementioned target speech synthesis model, and the speech synthesis method includes: The predicted text is obtained and input into the target speech synthesis model for model prediction to obtain the target synthesized speech.

[0014] Furthermore, to achieve the above objectives, this application also proposes a speech synthesis model training device, which includes: The conversion module is used to convert the punctuation marks in the input text into preset punctuation marks of the corresponding types to obtain text samples; The processing module is used to input the text sample into the initial speech synthesis model, process the text sample through the text encoder of the initial speech synthesis model, and output the text latent vector. The text latent vector includes the punctuation latent vector corresponding to the punctuation marks in the text sample. The initial speech synthesis model does not include an explicit prosody prediction module and a related explicit prosody label module. The training module is used to train the initial speech synthesis model based on the text latent vectors and the preset speech prosody mapping relationship to obtain the target speech synthesis model.

[0015] In addition, to achieve the above objectives, this application also proposes a speech synthesis model training device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the speech synthesis model training method described above.

[0016] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the speech synthesis model training method described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: Text samples are obtained by converting punctuation marks in the input text into corresponding preset punctuation marks. These samples are then input into an initial speech synthesis model, where the text encoder processes them to output latent text vectors. These latent text vectors include the punctuation latent vectors corresponding to the punctuation marks in the text samples. The initial speech synthesis model does not include an explicit prosodic prediction module or a related explicit prosodic label module. The initial speech synthesis model is then trained based on these latent text vectors and preset prosodic mapping relationships to obtain a target speech synthesis model. By removing explicit prosodic labels from traditional speech synthesis models, the amount of data and parameter size that the model needs to process is reduced. An implicit prosodic speech synthesis model is established using the prosodic information implied by punctuation marks, indirectly modeling prosodic features through punctuation marks. This achieves extreme lightweighting of the model while maintaining the naturalness of speech synthesis, significantly reducing the model's computational power and memory usage, making it adaptable to low-computing edge devices in low-end vehicles, and reducing the resource consumption of the speech synthesis model. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the first embodiment of the speech synthesis model training method of this application. Figure 2 This is a schematic diagram of the module structure of the speech synthesis model training device according to an embodiment of this application; Figure 3 This is a schematic diagram of the hardware operating environment involved in the speech synthesis model training method in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of this application embodiment is as follows: punctuation marks in the input text are converted into corresponding preset punctuation marks to obtain text samples; the text samples are input into an initial speech synthesis model, and the text encoder of the initial speech synthesis model processes the text samples to output text latent vectors, wherein the text latent vectors include punctuation latent vectors corresponding to the punctuation marks in the text samples, and the initial speech synthesis model does not include an explicit prosody prediction module and a related explicit prosody label module; the initial speech synthesis model is trained based on the text latent vectors and preset speech prosody mapping relationships to obtain a target speech synthesis model.

[0025] In this embodiment, for ease of description, the speech synthesis model training device will be used as the execution subject in the following description.

[0026] The VITS model is an end-to-end speech synthesis model that combines variational autoencoders (VAEs), normalized flow, and generative adversarial networks (GANs). Its core advantage lies in its ability to generate natural and fluent speech, and it has been widely used in various voice interaction scenarios, including in-vehicle voice systems. Prosody, as a suprasegmental feature in speech that transcends the level of individual phonemes, directly determines the rhythm, intonation, and naturalness of speech. Existing VITS models typically include an independent prosody prediction model, employing an explicit prosody control approach. Prosody modeling is achieved by predicting specific prosodic tags (such as duration and pitch markers) to ensure the expressiveness of the synthesized speech.

[0027] With the increasing prevalence of in-vehicle voice systems, the demand for speech synthesis functions in low-end vehicles is growing. However, due to cost constraints, the on-board devices in low-end vehicles suffer from low computing power and limited memory resources. This leads to frequent access to off-chip memory during model deployment, increasing latency and energy consumption, and potentially causing issues such as lag and startup failures. The explicit prosody prediction module of existing VITS models consumes significant computing power and memory resources due to the need to model, store, and compute a large number of explicit prosody labels. This makes traditional VITS models unsuitable for the low-computing-power on-board deployment requirements of low-end vehicles, limiting their widespread application in low-end in-vehicle scenarios.

[0028] Currently, lightweight improvements to the VITS model mainly focus on weight pruning, quantization, or inference engine optimization. These improvements cannot fundamentally solve the problem of excessive computing power and memory consumption caused by the prosody prediction module, making it difficult to meet the extreme lightweight requirements of low-end vehicle deployment.

[0029] Therefore, how to reduce the resource consumption of speech synthesis models is an urgent problem to be solved.

[0030] This application provides a solution that reduces the amount of data and parameter size that the model needs to process by removing explicit prosodic tags from traditional speech synthesis models. It also establishes an implicit prosodic speech synthesis model by using punctuation marks to implicitly model prosodic features. By indirectly modeling prosodic features through punctuation marks, the model achieves extreme lightweighting while ensuring the naturalness of speech synthesis. This significantly reduces the model's computing power and memory usage, making it compatible with low-computing-power edge devices in low-end vehicles and reducing the resource consumption of the speech synthesis model.

[0031] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or speech synthesis model training device capable of performing the above functions. The following description uses a speech synthesis model training device as an example to illustrate this embodiment and the subsequent embodiments.

[0032] Based on this, embodiments of this application provide a method for training a speech synthesis model, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the speech synthesis model training method of this application.

[0033] In this embodiment, the speech synthesis model training method includes steps S110~S130: Step S110: Convert the punctuation marks in the input text into preset punctuation marks of the corresponding types to obtain a text sample; In this embodiment, the initial speech synthesis model can be an improved VITS model. Specifically, the initial speech synthesis model includes a text encoder, a posterior encoder, a normalized stream, a random duration predictor, a decoder, and a discriminator, but does not include an explicit prosodic prediction module and a related explicit prosodic label module. In other words, this application removes the explicit prosodic labels and explicit prosodic prediction module from the VITS model and introduces a punctuation integration module to establish an implicit prosodic model, i.e., the initial speech synthesis model. This improves the prosodic modeling part of the traditional speech synthesis model, thereby reducing the model's computational power and memory usage from the source. An implicit prosodic model is established based on the integrated punctuation marks, achieving implicit modeling of prosodic information without the need for additional explicit prosodic prediction calculations and label storage.

[0034] In this embodiment of the application, the correspondence between each type of punctuation mark and the preset punctuation mark can be preset. The input text is preprocessed according to the correspondence to convert the punctuation marks in the input text into the corresponding preset punctuation marks to obtain a text sample.

[0035] In this embodiment, the number of preset punctuation marks can be reasonably set. For example, five preset punctuation marks can be set: period (.), exclamation mark (!), comma (,), question mark (?), and tilde (~), which correspond to terminating punctuation marks, exclamatory punctuation marks, pause punctuation marks, interrogative punctuation marks, and tone punctuation marks, respectively. That is, the types of preset punctuation marks include terminating, exclamatory, pause, interrogative, and tone punctuation marks. Therefore, the punctuation integration process includes: integration of terminating punctuation marks, integration of exclamatory punctuation marks, integration of pause punctuation marks, integration of interrogative punctuation marks, and integration of tone punctuation marks.

[0036] Specifically, the integration of terminating punctuation marks involves converting all terminating punctuation marks in the input text, such as periods (.), semicolons (;), colons (:), ellipses (...), and dashes (—), which indicate the end of a sentence or pause, into the preset punctuation mark period (). The integration of exclamatory punctuation marks specifically involves converting exclamation marks (!), double exclamation marks (!!), and other exclamation-related punctuation marks in the input text that indicate exclamation into the preset exclamation mark (!). The integration of pause punctuation marks specifically involves converting commas (,), pause marks (、), and other pause punctuation marks that indicate short pauses within sentences in the input text into commas (,) from the preset punctuation marks. The integration of interrogative punctuation marks specifically involves converting interrogative punctuation marks such as question marks (?), double question marks (??), and mixed interrogative and exclamatory punctuation marks (?!) in the input text into the preset question mark (?). The integration of tone-related punctuation marks specifically involves converting tone-related punctuation marks such as tilde (~) and ellipsis (...) in the input text, which indicate a gentler or longer tone, into tilde (~) in the preset punctuation marks.

[0037] It should be noted that for text fragments without punctuation marks in the input text, the corresponding preset punctuation marks will be automatically added according to the semantic logic of the sentence to ensure that each sentence fragment contains preset punctuation marks to imply rhythmic information. Among them, commas (,) or periods (.) can be added first.

[0038] In one feasible implementation, step S110 may include steps A110~A120: Step A110: Input the input text into the initial speech synthesis model; Step A120: The input text is preprocessed by the punctuation integration module of the initial speech synthesis model to convert the punctuation marks in the text into corresponding preset punctuation marks, thereby obtaining a text sample.

[0039] In this embodiment, a punctuation integration module can be set in the initial speech synthesis model. The punctuation integration module preprocesses the input text. Specifically, the input text is first input into the initial speech synthesis model, and the punctuation integration module preprocesses the input text to convert the punctuation marks in the text into corresponding preset punctuation marks, thereby obtaining text samples. That is, each punctuation mark in the input text is converted into a period (.), exclamation mark (!), comma (,), question mark (?), and tilde (~), etc., respectively, to obtain text samples.

[0040] In one feasible implementation, step S110 may include steps A130~A140: Step A130: Obtain each punctuation mark in the input text; Step A140: Based on the preset conversion relationship, convert each punctuation mark in the input text into a preset punctuation mark of the corresponding type to obtain a text sample.

[0041] In this embodiment, the input text can be preprocessed first through the punctuation integration module, and then the preprocessed text sample can be input into the initial speech synthesis model for model training.

[0042] Specifically, the input text is entered into the punctuation integration module, which then obtains each punctuation mark from the input text.

[0043] Next, based on the preset conversion relationship, each punctuation mark in the input text is converted into the corresponding preset punctuation mark to obtain a text sample. The preset conversion relationship is the conversion relationship corresponding to the integration of terminating punctuation, exclamatory punctuation, pause punctuation, interrogative punctuation, and tone punctuation. Then, the text sample is input into the initial speech synthesis model to train the model on the text sample.

[0044] Step S120: Input the text sample into the initial speech synthesis model, process the text sample through the text encoder of the initial speech synthesis model, and output the text latent vector. The text latent vector includes the punctuation latent vector corresponding to the punctuation marks in the text sample. The initial speech synthesis model does not include an explicit prosody prediction module and a related explicit prosody label module. In this embodiment, the text encoder of the initial speech synthesis model is improved in advance. During the text encoding process of the text encoder, the weight of the encoding features corresponding to punctuation marks is enhanced, so that the text encoder can focus on capturing the position and type information of punctuation marks.

[0045] Specifically, the text sample is input into the initial speech synthesis model, and the text sample is processed by the text encoder of the initial speech synthesis model. The text latent vectors output by the text encoder are used to obtain the vectors corresponding to the characters and punctuation marks in the text sample.

[0046] In one feasible implementation, the text latent vector includes the punctuation latent vector corresponding to the punctuation marks, and step S120 may include steps B110~B130: Step B110: The text sample is processed by the text encoder to obtain the text latent vector; Step B120: Increase the weight of the encoded features corresponding to punctuation marks in the text sample through the text encoder, and capture the position and type information of the punctuation marks in the text sample; Step B130: The text encoder determines the punctuation latent vector based on the location information and type information.

[0047] In this embodiment, the text latent vector may include punctuation latent vector and text latent vector. For the text latent vector, the text latent vector can be generated by the generation process of the speech synthesis model (VITS model) in related technologies. The text latent vector may include semantic latent vector, syntactic latent vector, etc.

[0048] Because the text encoder enhances the weight of the encoding features corresponding to punctuation marks during the text encoding process, the text sample is processed by the text encoder to increase the weight of the encoding features corresponding to punctuation marks in the text sample, and to capture the position and type information of the punctuation marks in the text sample.

[0049] Next, the text encoder determines the punctuation latent vectors corresponding to the punctuation marks in the text samples based on the position information and type information. For example, the punctuation latent vectors are determined according to the type information of the punctuation marks, and the position of the punctuation latent vectors in the text latent vectors is determined according to the position information, so as to integrate the punctuation marks in the text samples into the text latent vector representation, thereby realizing the implicit embedding of prosodic information without the need to output explicit prosodic labels.

[0050] Step S130: Based on the text latent vector and the preset speech prosody mapping relationship, the initial speech synthesis model is trained to obtain the target speech synthesis model.

[0051] The preset speech prosody mapping relationship includes the speech prosody association between each preset punctuation mark and speech prosody features.

[0052] In this embodiment of the application, a preset speech prosody mapping relationship can be established in the initial speech synthesis model so that, during the model training stage, preset punctuation marks and speech prosody features can be associated and trained according to the preset speech prosody mapping relationship, so that the model can learn the prosody patterns corresponding to different punctuation marks.

[0053] Among them, the preset speech prosody mapping relationship includes the speech prosody association relationship between each preset punctuation mark and speech prosody features. For example, the period (.) corresponds to a long pause at the end of a sentence, the comma (,) corresponds to a short pause within a sentence, the exclamation mark (!) corresponds to an upward tone and increased volume, the question mark (?) corresponds to a change in intonation in questioning tone, and the tilde (~) corresponds to a relaxed tone and a prolonged ending sound.

[0054] In this embodiment, the initial speech synthesis model is trained based on the text latent vector and the preset speech prosody mapping relationship to obtain the target speech synthesis model. Specifically, during the training process, the focus is on optimizing the accuracy of implicit prosody modeling and the naturalness of speech synthesis. The model parameters are adjusted through adversarial training (discriminator feedback) to ensure that the prosodic naturalness of the synthesized speech is not lower than that of the traditional speech synthesis model after the explicit prosodic labels are removed.

[0055] In one feasible implementation, step S130 may include step C110: Step C110: The initial speech synthesis model is trained based on the text latent vector and the preset speech prosody mapping relationship, and the phoneme duration distribution in the random duration predictor of the initial speech synthesis model is dynamically adjusted based on the punctuation marks in the text samples to obtain the target speech synthesis model.

[0056] In this embodiment, the initial speech synthesis model is trained based on text latent vectors and preset speech prosody mapping relationships. At the same time, a random duration predictor is used for coordinated training, and the phoneme duration distribution in the random duration predictor of the initial speech synthesis model is dynamically adjusted based on the punctuation marks in the text samples. This allows the random duration predictor to combine the implicit prosodic features corresponding to the punctuation marks when predicting phoneme duration, ensuring the prosodic naturalness of the synthesized speech and replacing the guiding role of explicit prosodic labels for duration prediction.

[0057] In this embodiment, by removing explicit prosody tags and the explicit prosody prediction module from the target speech synthesis model, the number of parameters in the model is reduced by 30%-50%, memory usage is reduced by 40%-60%, and computing power consumption is reduced by 35%-55% (experimental verification: the memory usage of the traditional speech synthesis model deployed on the edge is about 380-420MB, while the memory usage of the improved model of this invention can be reduced to 150-220MB; the computing power consumption of the traditional model for a single inference is about 2.5s, while the improved model can be reduced to 1.1-1.6s). It can perfectly adapt to the edge devices of low-end vehicles with low computing power and small memory, solve the pain point that the traditional speech synthesis model cannot be deployed on the edge of low-end vehicles, break through its application scenario limitations, and at the same time reduce the frequency of off-chip memory access, reduce latency and energy consumption, and significantly reduce computing power and memory usage.

[0058] By implicitly modeling prosody using punctuation marks to replace explicit prosody prediction, the model significantly reduces its size while ensuring the naturalness of the prosody in speech synthesis. According to the MOS (Mean Opinion Score) test, the improved model's MOS score is no less than 4.2 (out of 5), which is basically on par with the traditional speech synthesis model (MOS score of around 4.3), with no significant perceptual difference. This meets the usage requirements of in-vehicle voice interaction and solves the industry problem of "the incompatibility between lightweight and naturalness".

[0059] By optimizing only the prosodic modeling part of the speech synthesis model while retaining its original core architecture and interfaces, no modifications to the edge device hardware or the redevelopment of the entire speech synthesis system are required. This results in strong compatibility and rapid adaptation to existing low-end vehicle deployment processes, reducing deployment costs and development cycles for in-vehicle voice systems. Furthermore, the punctuation integration strategy standardizes the input text format, reducing prosodic modeling interference from different punctuation types. The implicit prosodic model has a simpler structure and is more adaptable to low-computing-power devices, making it suitable not only for low-end vehicles but also for other low-computing-power voice interaction scenarios (such as inexpensive smart speakers and portable voice devices), broadening its application scope. Additionally, removing explicit prosodic labels eliminates the need to process large amounts of prosodic label data during model training, reducing training data volume by 20%-30% and shortening the training cycle by 15%-25%, thus reducing training costs and time.

[0060] This embodiment provides a method for training a speech synthesis model. It obtains text samples by converting punctuation marks in the input text into corresponding preset punctuation marks. The text samples are then input into an initial speech synthesis model, where the text encoder processes them to output latent text vectors. These latent text vectors include punctuation latent vectors corresponding to the punctuation marks in the text samples. The initial speech synthesis model does not include an explicit prosodic prediction module or a related explicit prosodic label module. The initial speech synthesis model is then trained based on these latent text vectors and preset prosodic mapping relationships to obtain a target speech synthesis model. By removing explicit prosodic labels from traditional speech synthesis models, the amount of data and parameter size that the model needs to process is reduced. An implicit prosodic speech synthesis model is established using the prosodic information implied by punctuation marks. By indirectly modeling prosodic features through punctuation marks, the model achieves extreme lightweighting while maintaining the naturalness of speech synthesis. This significantly reduces the model's computational power and memory usage, making it adaptable to low-computing edge devices in low-end vehicles and reducing the resource consumption of the speech synthesis model.

[0061] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the speech synthesis model training method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0062] This application also provides a speech synthesis method.

[0063] The speech synthesis method is applied to a vehicle, which is equipped with the target speech synthesis model described in the foregoing embodiment. The speech synthesis method includes step S210: Step S210: Obtain the predicted text and input the predicted text into the target speech synthesis model for model prediction to obtain the target synthesized speech.

[0064] In this embodiment, after obtaining the target speech synthesis model, the target speech synthesis model can be deployed to the edge device of a low-end vehicle. For example, after lightweight optimization such as quantization and operator fusion, the optimized target speech synthesis model can be deployed to the edge device of a low-end vehicle. Since the model removes the explicit prosody prediction module, the parameter scale and computational load are greatly reduced, which can be adapted to the low computing power and small memory hardware environment of low-end vehicles. It does not need to frequently access off-chip memory, reducing latency and energy consumption, thereby making the target speech synthesis model applicable to all vehicle models.

[0065] When speech synthesis is required, a predicted text is obtained. If the target speech synthesis model includes a symbol integration module, the predicted text is directly input into the target speech synthesis model for model training to obtain the target synthesized speech; that is, the predicted text is the text without punctuation integration. If the target speech synthesis model does not include a symbol integration module, a symbol integration module needs to be deployed. The text to be predicted is input into the symbol integration module to integrate punctuation marks to obtain the predicted text, which is then input into the target speech synthesis model for model training to obtain the target synthesized speech; that is, the predicted text is the text with punctuation integration.

[0066] This embodiment provides a speech synthesis method. By acquiring predicted text and inputting the predicted text into the target speech synthesis model for model prediction, the target synthesized speech is obtained. By removing explicit prosodic tags in traditional speech synthesis models, the amount of data and parameter scale that the model needs to process is reduced. Then, the target synthesized speech is obtained by prediction through the target speech synthesis model. While ensuring the naturalness of speech synthesis, the model achieves extreme lightweighting, significantly reducing the model's computing power and memory usage, making it adaptable to low-computing-power edge devices in low-end vehicles, and reducing the resource consumption of the speech synthesis model.

[0067] This application also provides a speech synthesis model training device, please refer to... Figure 2 The speech synthesis model training device includes: The conversion module 10 is used to convert the punctuation marks in the input text into preset punctuation marks of the corresponding types to obtain text samples; Processing module 20 is used to input the text sample into the initial speech synthesis model, process the text sample through the text encoder of the initial speech synthesis model, and output text latent vectors, wherein the text latent vectors include the punctuation latent vectors corresponding to the punctuation marks in the text sample, and the initial speech synthesis model does not include an explicit prosody prediction module and a related explicit prosody label module. Training module 30 is used to train the initial speech synthesis model based on the text latent vector and the preset speech prosody mapping relationship to obtain the target speech synthesis model.

[0068] The speech synthesis model training device provided in this application, employing the speech synthesis model training method in the above embodiments, can solve the technical problem of how to reduce the resource consumption of the speech synthesis model. Compared with the prior art, the beneficial effects of the speech synthesis model training device provided in this application are the same as those of the speech synthesis model training method provided in the above embodiments, and other technical features in the speech synthesis model training device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0069] This application provides a speech synthesis model training device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the speech synthesis model training method in Embodiment 1 above.

[0070] The following is for reference. Figure 3 The diagram illustrates a structural schematic suitable for implementing a speech synthesis model training device according to embodiments of this application. The speech synthesis model training device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 3 The speech synthesis model training device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0071] like Figure 3 As shown, the speech synthesis model training device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the speech synthesis model training device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the speech synthesis model training device to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows speech synthesis model training devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0072] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0073] The speech synthesis model training device provided in this application, employing the speech synthesis model training method in the above embodiments, can solve the technical problem of how to reduce the resource consumption of the speech synthesis model. Compared with the prior art, the beneficial effects of the speech synthesis model training device provided in this application are the same as those of the speech synthesis model training method provided in the above embodiments, and other technical features in this speech synthesis model training device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0074] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0075] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0076] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the speech synthesis model training method in the above embodiments.

[0077] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0078] The aforementioned computer-readable storage medium may be included in the speech synthesis model training device; or it may exist independently and not be assembled into the speech synthesis model training device.

[0079] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a speech synthesis model training device, the speech synthesis model training device: converts punctuation marks in the input text into corresponding preset punctuation marks to obtain text samples; inputs the text samples into an initial speech synthesis model, processes the text samples through the text encoder of the initial speech synthesis model, and outputs text latent vectors, wherein the text latent vectors include punctuation latent vectors corresponding to the punctuation marks in the text samples, and the initial speech synthesis model does not include an explicit prosody prediction module and a related explicit prosody label module; and trains the initial speech synthesis model based on the text latent vectors and preset speech prosody mapping relationships to obtain a target speech synthesis model.

[0080] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0081] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0082] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0083] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described speech synthesis model training method, thereby solving the technical problem of how to reduce the resource consumption of speech synthesis models. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the speech synthesis model training method provided in the above embodiments, and will not be repeated here.

[0084] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the speech synthesis model training method described above.

[0085] The computer program product provided in this application solves the technical problem of how to reduce the resource consumption of speech synthesis models. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the speech synthesis model training method provided in the above embodiments, and will not be repeated here.

[0086] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A method for training a speech synthesis model, characterized in that, The speech synthesis model training method includes: The punctuation marks in the input text are converted into corresponding preset punctuation marks to obtain text samples; The text sample is input into the initial speech synthesis model, and the text encoder of the initial speech synthesis model processes the text sample to output a text latent vector. The text latent vector includes the punctuation latent vectors corresponding to the punctuation marks in the text sample. The initial speech synthesis model does not include an explicit prosody prediction module and a related explicit prosody label module. The initial speech synthesis model is trained based on the text latent vectors and the preset speech prosody mapping relationship to obtain the target speech synthesis model.

2. The speech synthesis model training method as described in claim 1, characterized in that, The text latent vector includes the text latent vectors corresponding to the characters in the text sample. The step of processing the text sample through the text encoder of the initial speech synthesis model and outputting the text latent vector includes: The text sample is processed by the text encoder to obtain the text latent vector; The text encoder increases the weight of the encoded features corresponding to punctuation marks in the text sample and captures the position and type information of the punctuation marks in the text sample. The text encoder determines the punctuation latent vector based on the location and type information.

3. The speech synthesis model training method as described in claim 1, characterized in that, The preset speech prosody mapping relationship includes the speech prosody association between each preset punctuation mark and speech prosody features.

4. The speech synthesis model training method as described in claim 1, characterized in that, The step of training the target speech synthesis model based on the text latent vectors and the preset speech prosody mapping relationship through the initial speech synthesis model includes: The initial speech synthesis model is trained based on the text latent vectors and the preset speech prosody mapping relationship. The phoneme duration distribution in the random duration predictor of the initial speech synthesis model is dynamically adjusted based on the punctuation marks in the text samples to obtain the target speech synthesis model.

5. The speech synthesis model training method according to any one of claims 1 to 4, characterized in that, The step of converting the punctuation marks in the input text into corresponding preset punctuation marks to obtain a text sample includes: Input the input text into the initial speech synthesis model; The input text is preprocessed by the punctuation integration module of the initial speech synthesis model to convert the punctuation marks in the text into preset punctuation marks of the corresponding type, thereby obtaining a text sample.

6. The speech synthesis model training method according to any one of claims 1 to 4, characterized in that, The step of converting the punctuation marks in the input text into corresponding preset punctuation marks to obtain a text sample includes: Obtain each punctuation mark in the input text; Based on the preset conversion relationship, each punctuation mark in the input text is converted into a preset punctuation mark of the corresponding type to obtain a text sample.

7. A speech synthesis method, characterized in that, Applied to a vehicle, the vehicle is equipped with a target speech synthesis model according to any one of claims 1 to 6, the speech synthesis method comprising: The predicted text is obtained and input into the target speech synthesis model for model prediction to obtain the target synthesized speech.

8. A speech synthesis model training device, characterized in that, The speech synthesis model training device includes: The conversion module is used to convert the punctuation marks in the input text into preset punctuation marks of the corresponding types to obtain text samples; The processing module is used to input the text sample into the initial speech synthesis model, process the text sample through the text encoder of the initial speech synthesis model, and output the text latent vector. The text latent vector includes the punctuation latent vector corresponding to the punctuation marks in the text sample. The initial speech synthesis model does not include an explicit prosody prediction module and a related explicit prosody label module. The training module is used to train the initial speech synthesis model based on the text latent vectors and the preset speech prosody mapping relationship to obtain the target speech synthesis model.

9. A speech synthesis model training device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the speech synthesis model training method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the speech synthesis model training method as described in any one of claims 1 to 7.

Images