Speech synthesis methods, systems, electronic devices and storage media

Abstract

This invention provides a speech synthesis method, system, electronic device, and storage medium. The method includes: acquiring a target speech synthesis instruction and a target speech synthesis text; performing structured processing on the target speech synthesis instruction based on annotation information of the target speech synthesis instruction to obtain a structured instruction; the annotation information includes word segmentation information, named entity information, and phoneme information; and performing speech synthesis based on the structured instruction and the target speech synthesis text to obtain synthesized speech. This invention enhances the understanding and parsing ability of implicit information in free instructions to dynamically generate corresponding speech synthesis parameters, thereby generating synthesized speech that better meets user expectations and effectively improving the accuracy and consistency of synthesized speech.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a speech synthesis method, system, electronic device, and storage medium. Background Technology

[0002] In the field of natural language processing, instruction-based speech synthesis technology has been widely used because it can flexibly control various attributes of synthesized speech, such as gender, age, timbre, emotion, and speech rate, and allows users or developers to guide the speech synthesis process through explicit instructions, thereby generating speech output that meets specific needs.

[0003] Currently, in the process of speech synthesis, speech synthesis models are usually used to parse structured, predefined, standardized instruction information to obtain key synthesis information, and then speech synthesis is performed on this key synthesis information to generate speech output that meets the instruction requirements.

[0004] In practical applications, user input often lacks structured information, the selection and description of command words are not precise enough, and non-standardized command information that often omits key information makes it difficult for current speech synthesis models to accurately parse and understand commands, resulting in the generated speech results not matching user expectations, and thus affecting the accuracy and consistency of synthesized speech. Summary of the Invention

[0005] This invention provides a speech synthesis method, system, electronic device, and storage medium to address the shortcomings of poor accuracy and consistency in synthesized speech in existing technologies. It improves the accuracy and consistency of synthesized speech by enhancing the model's ability to understand and parse implicit information in non-standardized instruction information.

[0006] This invention provides a speech synthesis method, comprising:

[0007] Obtain the target speech synthesis instruction and the target speech synthesis text;

[0008] Based on the annotation information of the target speech synthesis instruction, the target speech synthesis instruction is subjected to structured processing to obtain a structured instruction; the annotation information includes word segmentation information, named entity information, and phoneme information;

[0009] Speech synthesis is performed based on the structured instructions and the target speech text to obtain synthesized speech.

[0010] According to the present invention, a speech synthesis method is provided, wherein the step of performing structured processing on the target speech synthesis instruction based on the annotation information of the target speech synthesis instruction to obtain a structured instruction includes:

[0011] The annotation information of the target speech synthesis instruction is input into the instruction conversion model to obtain the structured instruction;

[0012] The instruction conversion model is obtained by training a pre-trained language model based on sample speech synthesis instructions in different scenarios and the structured labels corresponding to each sample speech synthesis instruction.

[0013] According to the present invention, a speech synthesis method is provided, wherein the training steps of the instruction conversion model include:

[0014] Based on the annotation information of each sample speech synthesis instruction and the structured label corresponding to each sample speech synthesis instruction, the pre-trained language model is iteratively trained to obtain the first language model;

[0015] If it is found that the first language model fails the performance verification, the structure and parameters of the first language model are adjusted according to the performance verification results of the first language model to obtain the second language model.

[0016] Based on each sample speech synthesis instruction and the sample label corresponding to each sample speech synthesis instruction, the second language model is iteratively trained until a trained second language model that passes performance verification is obtained.

[0017] The instruction conversion model is constructed based on the trained second language model that has passed performance verification;

[0018] The sample labels include structured labels and labels corresponding to language processing tasks, and the language processing tasks include question-answering tasks and / or translation tasks.

[0019] According to the present invention, a speech synthesis method is provided, wherein iterative training of a second language model is performed based on each sample speech synthesis instruction and the sample label corresponding to each sample speech synthesis instruction, comprising:

[0020] Each of the sample speech synthesis instructions is subjected to instruction enhancement processing to obtain each enhanced speech synthesis instruction;

[0021] The second language model is iteratively trained based on each of the enhanced speech synthesis instructions and the sample labels corresponding to each of the enhanced speech synthesis instructions;

[0022] The instruction enhancement process includes at least one of synonym replacement, word order reordering, and structural reordering.

[0023] According to the present invention, a speech synthesis method is provided, wherein the step of obtaining the annotation information of each sample speech synthesis instruction includes:

[0024] Each of the sample speech synthesis instructions is subjected to a first preprocessing to obtain preprocessed sample speech synthesis instructions;

[0025] A second preprocessing is performed on each preprocessed sample speech synthesis instruction to obtain the annotation information of each sample speech synthesis instruction;

[0026] The first preprocessing includes data cleaning and / or text planning, and includes word segmentation, named entity recognition, and phoneme annotation.

[0027] According to the present invention, a speech synthesis method is provided, wherein the step of performing speech synthesis based on the structured instructions and the target speech synthesis text to obtain synthesized speech includes:

[0028] Based on the instruction encoder, the structured instructions are segmented and encoded to obtain instruction features;

[0029] Based on a text encoder, the target speech-synthesized text is segmented and encoded to obtain text features;

[0030] Based on the instruction features and the text features, obtain the speech features;

[0031] The synthesized speech is obtained by decoding the speech features.

[0032] The model structure of the instruction encoder is the same as that of the text encoder.

[0033] According to the present invention, a speech synthesis method is provided, wherein obtaining speech features based on the instruction features and the text features includes:

[0034] Based on a preset delimiter, the instruction features and the text features are concatenated to obtain the target input information;

[0035] The target input information is input into a large language model to obtain the speech features;

[0036] The large-scale language model is used to perform semantic understanding on the target input information, obtain the instruction intent and content information, and synthesize the speech features based on the instruction intent and the content information.

[0037] The present invention also provides a speech synthesis system, comprising:

[0038] Obtain the target speech synthesis instruction and the target speech synthesis text;

[0039] Based on the annotation information of the target speech synthesis instruction, the target speech synthesis instruction is subjected to structured processing to obtain a structured instruction; the annotation information includes word segmentation information, named entity information, and phoneme information;

[0040] Speech synthesis is performed based on the structured instructions and the target speech text to obtain synthesized speech.

[0041] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described speech synthesis methods.

[0042] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech synthesis method as described above.

[0043] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described speech synthesis methods.

[0044] The speech synthesis method, system, electronic device, and storage medium provided by this invention perform structured processing on the target speech synthesis instructions based on the annotation information of the target speech synthesis instructions. This enhances the understanding and parsing of implicit information in the target speech synthesis instructions by utilizing annotation information such as word segmentation, named entities, and phonemes in the annotation information. In turn, it captures potential semantic elements from incomplete instructions, thereby more accurately identifying potential and standardized speech quality attributes in the target speech synthesis instructions, obtaining structured instructions, and controlling the generation of synthesized speech based on structured instructions and target speech synthesis text. This effectively improves the ability to accurately parse and understand the implicit information in the instructions, thereby generating synthesized speech that better meets user expectations and enhancing the accuracy and consistency of synthesized speech. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0046] Figure 1 This is one of the flowcharts of the speech synthesis method provided by the present invention.

[0047] Figure 2 This is the second flowchart of the speech synthesis method provided by the present invention.

[0048] Figure 3 This is a schematic diagram of the speech synthesis system provided by the present invention.

[0049] Figure 4This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] In the field of speech synthesis, the fundamental properties of speech synthesis technology rely on information provided by existing data, such as gender, age, timbre, and emotion. This information provides a solid foundation for generating natural and expressive speech.

[0052] In existing speech synthesis frameworks, the synthesis process is typically guided by predefined instructions. These instructions often contain explicit and crucial synthesis parameters, such as speech rate and emotional intensity, which help the model accurately understand and execute the speech synthesis task. In this case, the synthesis model can efficiently generate speech that meets expectations, ensuring the naturalness and controllability of the sound. The effectiveness of this process depends on the clarity and completeness of the instruction information.

[0053] Therefore, current command speech synthesis technology typically uses speech synthesis models to parse structured, predefined, standardized command information to obtain key synthesis information such as emotion, timbre, and speech rate. Then, it performs speech synthesis based on this key synthesis information to generate speech output that meets the command requirements, that is, to ensure that the synthesized speech can accurately express the speech attributes required by the user.

[0054] These speech synthesis methods exhibit high accuracy when processing explicit commands because the models can generate speech results with specific timbre and emotion based on clear parameter configurations. Technically, these methods typically utilize pre-set corpora and specific labeling systems, training the model through supervised learning to extract key synthesis parameters from the label information in the commands. This structured approach significantly simplifies the speech synthesis process and improves the stability and consistency of the generated speech.

[0055] However, existing speech synthesis methods often face significant limitations when dealing with free user commands—that is, non-standardized commands lacking structured information, with imprecise word selection and description, and often missing key information. Specifically, free commands lack structured information, the selection and description of command words are imprecise, and key information is often omitted. This uncertainty makes it difficult for existing synthesis models to accurately parse and understand commands, resulting in generated speech that does not match user expectations. For example, the user may only provide a vague emotional description without clearly indicating the intensity or details of the emotion, leading to unsatisfactory synthesized speech. Furthermore, without explicit labels, the model is prone to bias when extracting effective information from vague commands, thus affecting the accuracy and consistency of the synthesized speech.

[0056] To address the shortcomings of existing technologies in synthesizing speech with limited accuracy and consistency under free commands, this application provides a speech synthesis method. This method enhances the understanding and parsing of implicit information in free commands to dynamically generate corresponding speech synthesis parameters, thereby improving the accuracy and consistency of synthesized speech.

[0057] It should be noted that the subject executing this method can be a speech synthesis system, which can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, super mobile personal computers, netbooks, or personal digital assistants, etc., while non-mobile electronic devices can be servers, network attached storage devices, personal computers, etc., and this invention does not impose specific limitations.

[0058] Figure 1 This is one of the flowcharts illustrating the speech synthesis method provided by the present invention, such as... Figure 1 As shown, the method includes steps 110, 120 and 130.

[0059] Step 110: Obtain the target speech synthesis instruction and the target speech synthesis text.

[0060] Here, the target speech-to-text represents the content text used to synthesize speech. The target speech-to-text can include one or more sentences, such as a paragraph or an entire text containing multiple paragraphs. The target speech-to-text can be directly input by the user, or it can be obtained by capturing images through image acquisition devices such as scanners, mobile phones, and cameras and performing optical character recognition on the images. Alternatively, it can be obtained by transcribing captured speech, or it can be obtained by crawling the Internet. This embodiment of the invention does not specifically limit these methods.

[0061] The target speech synthesis instruction is a free instruction that describes the desired speech quality attributes when synthesizing speech using natural language. These speech quality attributes include, but are not limited to, speech rate, emotional intensity, timbre, gender, age, and the environmental context in which the speech occurs; this embodiment does not specifically limit these attributes. The target speech synthesis instruction can be directly input by the user or obtained by collecting and transcribing user input; this embodiment does not specifically limit these attributes either.

[0062] It should be noted that the user input referred to can be information input through command line interface, graphical interface, touch input, drop-down selection input, voice input, gesture input, visual input, brain-computer input, etc. This embodiment does not specifically limit it.

[0063] Step 120: Based on the annotation information of the target speech synthesis instruction, perform structured processing on the target speech synthesis instruction to obtain a structured instruction; the annotation information includes word segmentation information, named entity information, and phoneme information.

[0064] Optionally, after obtaining the target speech synthesis instruction, the instruction can be annotated to obtain its annotation information. For example, natural language processing tools or models can be used to segment the instruction into words to obtain segmentation information, which helps in accurately understanding the instruction content. Additionally, named entity recognition can be performed to obtain named entity information, further aiding in accurate understanding. Furthermore, phoneme-level annotation can be applied to obtain the phoneme sequence, or phoneme information, for accurate understanding of the instruction's phoneme features. Subsequently, annotation information for the target speech synthesis instruction is constructed based on the segmentation information, named entity information, and phoneme information.

[0065] After obtaining the annotation information of the target speech synthesis instruction, the instruction can be structured based on this information to obtain a structured instruction. This structured instruction is a precise instruction corresponding to the target speech synthesis instruction, with clearly defined instruction words that reflect the actual speech quality attributes.

[0066] The structuring process here can be implemented through pre-defined logic or a pre-trained model. For example, it could involve template matching based on the annotation information of the target speech synthesis instruction, followed by structuring the instruction based on the matched template to obtain a structured instruction. Alternatively, the annotation information of the target speech synthesis instruction could be input into a pre-trained instruction conversion model, which would then perform structuring to obtain a structured instruction. This instruction conversion model could be formed by collecting a large number of free instructions describing the desired speech quality attributes of synthesized speech in natural language, creating sample speech synthesis instructions, and matching these sample instructions with corresponding structured and precise instructions to form structured labels. The model would then be trained based on the sample speech synthesis instructions and their corresponding structured labels, enabling the trained instruction conversion model to output corresponding structured and precise instructions for free instructions.

[0067] Step 130: Perform speech synthesis based on the structured instructions and the target speech synthesis text to obtain synthesized speech.

[0068] Optionally, after obtaining the structured instructions, the structured instructions and the target speech synthesis text can be combined to perform speech synthesis to obtain the corresponding synthesized speech. It should be understood that the synthesized speech here is speech whose speech content is consistent with the target speech synthesis text, and whose speech quality attributes are consistent with the synthesized speech quality attributes indicated by the target speech synthesis instructions.

[0069] Here, speech synthesis can be achieved through a pre-trained model; for example, the structured instructions and the target speech synthesis text can be directly input into the pre-trained language model to output the corresponding synthesized speech, or the structured instructions and the target speech synthesis text can be first feature-encoded and feature-fused respectively, and then the fused features can be input into the pre-trained language model to output the corresponding synthesized speech.

[0070] It should be noted that regardless of whether the target speech synthesis instruction contains one or more speech quality attributes, the synthesis model can accurately identify the speech quality attributes and thereby achieve speech synthesis for one or more controllable dimensions of speech quality attributes, so as to efficiently generate speech that meets expectations and ensure the naturalness, controllability and flexibility of the sound.

[0071] The method provided in this embodiment performs structured processing on the target speech synthesis instructions based on the annotation information of the target speech synthesis instructions. This enhances the understanding and parsing of implicit information in the target speech synthesis instructions by utilizing annotation information such as word segmentation, named entities, and phonemes in the annotation information. In turn, it captures potential semantic elements from incomplete instructions, thereby more accurately identifying potential and standardized speech quality attributes in the target speech synthesis instructions, obtaining structured instructions, and controlling the generation of synthesized speech based on structured instructions and target speech synthesis text. This effectively improves the ability to accurately parse and understand the implicit information in the instructions, thereby generating synthesized speech that better meets user expectations and enhancing the accuracy and consistency of synthesized speech.

[0072] In some embodiments, step 120 specifically includes:

[0073] The annotation information of the target speech synthesis instruction is input into the instruction conversion model to obtain the structured instruction;

[0074] The instruction conversion model is obtained by training a pre-trained language model based on sample speech synthesis instructions in different scenarios and the structured labels corresponding to each sample speech synthesis instruction.

[0075] Optionally, the structured instruction can be a structured, precise instruction with clear word descriptions that reflects the actual speech quality attributes, obtained by the instruction conversion model. In other words, the annotation information of the target speech synthesis instruction can be input into the instruction conversion model, so that the instruction conversion model can apply the annotation information to perform structured processing of the target speech synthesis instruction, and output the structured instruction of the target speech synthesis instruction.

[0076] The training steps of the instruction conversion model here include: collecting a large number of free instructions expressed by users in natural speech in different scenarios to form sample speech synthesis instructions; matching corresponding structured and precise instructions for each sample speech synthesis instruction, and representing them in a specific format or encoding language to ensure that each sample speech synthesis instruction has a corresponding structured label; then, preprocessing the sample speech synthesis instructions in different scenarios to obtain the annotation information of each sample speech synthesis instruction. Using the annotation information of each sample speech synthesis instruction as samples and the structured labels as labels, the pre-trained language model is jointly trained in single or multi-task mode, thereby obtaining instructions that can effectively output corresponding structured instructions with accurate speech quality attributes for free instructions, thus improving the flexibility and accuracy of the speech synthesis system.

[0077] The scenarios include, but are not limited to, daily life scenarios, office scenarios, entertainment scenarios, and smart home control scenarios. This embodiment does not specifically limit these scenarios.

[0078] The pre-trained language model can be constructed from various pre-trained language models suitable for instruction translation tasks, such as a model constructed using a fine-tuned language network with T5 (FLAN-T5) based on the T5 architecture. This embodiment does not specifically limit this. FLAN-T5 is a T5 model that has been fine-tuned for instruction translation, possessing powerful natural language understanding and generation capabilities, and is suitable for instruction translation tasks.

[0079] The method provided in this embodiment introduces a more adaptive natural language understanding mechanism from a pre-trained language model, enabling the instruction conversion model to capture potential semantic elements from incomplete instruction information and dynamically generate corresponding speech synthesis parameters. Simultaneously, it also introduces a more intelligent contextual reasoning mechanism from the pre-trained language model, helping the instruction conversion model more accurately identify the user's emotional intent and timbre preferences in the absence of explicit parameter labels. This improves the quality of synthesized speech in free-command scenarios, thereby better meeting the complex and diverse needs of users.

[0080] In some embodiments, the training steps of the instruction translation model include:

[0081] Based on the annotation information of each sample speech synthesis instruction and the structured label corresponding to each sample speech synthesis instruction, the pre-trained language model is iteratively trained to obtain the first language model;

[0082] If it is found that the first language model fails the performance verification, the structure and parameters of the first language model are adjusted according to the performance verification results of the first language model to obtain the second language model.

[0083] Based on each sample speech synthesis instruction and the sample label corresponding to each sample speech synthesis instruction, the second language model is iteratively trained until a trained second language model that passes performance verification is obtained.

[0084] The instruction conversion model is constructed based on the trained second language model that has passed performance verification;

[0085] The sample labels include structured labels and labels corresponding to language processing tasks, and the language processing tasks include question-answering tasks and / or translation tasks.

[0086] Optionally, the training steps of the instruction conversion model specifically include: model design steps and model fine-tuning steps, model validation steps and model optimization steps.

[0087] The model design process includes the following steps:

[0088] Step S10: Select the model architecture of the pre-trained language model, such as selecting the FLAN-T5 model architecture as the model architecture of the pre-trained language model.

[0089] Step S11: Based on computing resources and task requirements, select an appropriate model size, such as a small T5 model (T5-small), a medium-sized T5 model (T5-base), or a large-scale T5 model (T5-large).

[0090] Step S12: Use the model parameters of the selected model architecture as initial values ​​to construct a pre-trained language model.

[0091] The model fine-tuning process includes the following steps:

[0092] Step S20, Input and output format definition: Determine the input and output formats of the model, using the annotation information of the sample speech synthesis instructions (i.e., sample free instructions) as the input sequence and the structured labels as the target output sequence;

[0093] Step S21, Loss function setting: Based on the difference between the model output and the structured labels, construct a loss function, such as the cross-entropy loss function, etc. This embodiment does not specifically limit this;

[0094] Step S22, Fine-tuning Training: Based on the annotation information of the speech synthesis instructions of each sample in the training set, and the structured labels corresponding to the speech synthesis instructions of each sample, the pre-trained language model is trained in multiple rounds of iteration. In each round, all training data is traversed. After each training cycle, the model performance is evaluated on the validation set, and the changes in the loss function and evaluation index are monitored until the maximum number of iterations is reached, or the loss function converges or the evaluation index converges. Then, training is stopped, and the pre-trained language model with the best combined loss function and evaluation index during the training process is obtained to construct the first language model.

[0095] The model validation steps specifically include:

[0096] Performance metrics were calculated using the annotation information of each speech synthesis instruction in the test set, as well as the structured tags corresponding to each speech synthesis instruction, to verify and measure the instruction conversion performance of the first language model. These performance metrics included, but were not limited to, Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation (ROUGE), and accuracy.

[0097] If the first language model passes the verification, such as when the values ​​of each performance indicator meet their corresponding threshold, or when the fusion index values ​​of multiple performance indicators meet their corresponding fusion index thresholds, then the instruction conversion model is directly constructed based on the first language model.

[0098] If the first language model fails validation, such as when the value of any performance metric does not meet its corresponding threshold, or when the fusion metric value of multiple performance metrics does not meet its corresponding fusion metric threshold, the error conversion cases of the first language model can be analyzed to understand the reasons for the errors, such as comprehension errors or generation errors. Alternatively, based on the manual evaluation of the structured instructions output by professionals, the correctness and usability of the model can be examined to analyze the reasons for the first language model failing validation, so that the model can be further optimized and trained based on the first language model.

[0099] If the first language model fails the performance validation, the following model optimization steps will be performed:

[0100] Based on the performance verification results of the first language model, its structure and parameters are adjusted to obtain a second language model. Then, using the speech synthesis instructions and corresponding multi-task sample labels from the training set, the second language model is optimized and trained. The model verification and optimization steps are repeated until a trained second language model that passes performance verification is obtained. Training is then stopped to improve instruction conversion performance. The validated second language model is saved for future use in instruction conversion tasks, where it can be used to deploy and construct the final instruction conversion model for structured instruction processing.

[0101] Here, the optimized training of the second language model can be performed by directly training the second language model using the speech synthesis instructions of each sample and the multi-task sample labels corresponding to each speech synthesis instruction, or by first preprocessing the speech synthesis instructions of each sample, such as data augmentation, and then training the second language model based on the preprocessed speech synthesis instructions and the multi-task sample labels corresponding to the preprocessed speech synthesis instructions. This embodiment does not specifically limit this.

[0102] Here, the sample labels for multiple tasks include, but are not limited to, the labels corresponding to the instruction conversion task, i.e., structured labels, as well as the labels corresponding to the question answering task and / or the labels corresponding to the translation task. By jointly training the instruction conversion task with related natural language processing tasks (such as question answering and translation), the generalization ability of the model can be effectively improved.

[0103] The method provided in this embodiment first forms a preliminary first language model through iterative training based on a pre-trained language model. After verifying its performance, the structure and parameters are finely adjusted for the parts that do not meet the standards to obtain a second language model. Then, joint training is performed based on sample labels from multiple tasks, including instruction conversion, question answering, and translation, which significantly enhances the generalization ability of the model. Finally, the second language model, which has passed performance verification, is used to construct an instruction conversion model, realizing accurate and structured processing of sample speech synthesis instructions and effectively improving the accuracy and efficiency of instruction conversion.

[0104] In some embodiments, the iterative training of the second language model based on each of the sample speech synthesis instructions and the sample labels corresponding to each of the sample speech synthesis instructions includes:

[0105] Each of the sample speech synthesis instructions is subjected to instruction enhancement processing to obtain each enhanced speech synthesis instruction;

[0106] The second language model is iteratively trained based on each of the enhanced speech synthesis instructions and the sample labels corresponding to each of the enhanced speech synthesis instructions;

[0107] The instruction enhancement process includes at least one of synonym replacement, word order reordering, and structural reordering.

[0108] Optionally, the model optimization step further includes:

[0109] Synonym replacement is performed on each sample speech synthesis instruction (i.e., sample free instructions) to increase expressive diversity, and / or instruction recombination is performed on each sample speech synthesis instruction, i.e., changing the word order or structure of the instructions, to generate new training samples, thereby obtaining enhanced speech synthesis instructions. Then, based on the enhanced speech synthesis instructions and their corresponding sample labels, the second language model is iteratively trained, and the above model verification and optimization steps are repeated until a trained second language model that passes performance verification is obtained. Training then stops to improve instruction conversion performance. The trained second language model that has passed performance verification is saved for subsequent instruction conversion tasks, where it can be used to deploy and construct the final instruction conversion model for instruction structuring.

[0110] The method provided in this embodiment enhances the sample speech synthesis instructions by performing instruction enhancement processing, such as synonym replacement, word order reordering, and structural reordering, thereby increasing the diversity and coverage of training data. Based on this, the model is trained, which not only improves the model's adaptability to different expression methods, but also enhances the model's understanding of the latent semantics in the instructions. This enables the model to more accurately convert unstructured speech synthesis instructions into structured instructions, thereby improving the accuracy and consistency of synthesized speech and user satisfaction in practical applications.

[0111] In some embodiments, the step of obtaining the annotation information of each sample speech synthesis instruction includes:

[0112] Each of the sample speech synthesis instructions is subjected to a first preprocessing to obtain preprocessed sample speech synthesis instructions;

[0113] A second preprocessing is performed on each preprocessed sample speech synthesis instruction to obtain the annotation information of each sample speech synthesis instruction;

[0114] The first preprocessing includes data cleaning and / or text planning, and includes word segmentation, named entity recognition, and phoneme annotation.

[0115] Optionally, the steps for obtaining the annotation information of each sample speech synthesis instruction specifically include:

[0116] Step S00: Collect a large number of free commands expressed by users in natural language form to form sample speech synthesis commands;

[0117] Step S01 involves preprocessing the collected sample speech synthesis instructions, such as noise reduction, removal of silence segments, and normalization, as well as information annotation, to improve data quality. Specific implementation steps include:

[0118] Step S011 involves cleaning the sample speech synthesis instructions, specifically removing duplicate, incorrect, or incomplete instruction pairs to correct spelling and grammatical errors and ensure data quality.

[0119] Step S012 involves text normalization of the sample speech synthesis instructions, specifically including unified text normalization of the sample speech synthesis instructions and structured tags, such as lowercase conversion, punctuation processing, and special character escaping.

[0120] Step S013: The preprocessed sample speech synthesis instructions are segmented and labeled. Specifically, the preprocessed sample speech synthesis instructions are segmented and part-of-speech tagging or named entity recognition is performed as needed to help the model understand the instruction content.

[0121] Step S014: Phoneme annotation is performed on the preprocessed sample speech synthesis instructions. Specifically, this includes using professional annotation tools to annotate the preprocessed speech data at the phoneme level to generate the corresponding phoneme sequence, i.e., phoneme information.

[0122] By processing each sample speech synthesis instruction through the above steps, the annotation information of each sample speech synthesis instruction can be obtained. Subsequently, based on the annotation information of each sample speech synthesis instruction, the model can be iteratively trained to quickly and accurately obtain an instruction conversion model that can accurately output structured instructions based on free instructions.

[0123] The method provided in this embodiment can efficiently and accurately extract high-quality annotation information from sample free instructions. This information includes the processing results of data cleaning, text normalization, word segmentation, named entity recognition, and phoneme annotation. It provides rich and accurate training data for building a high-performance instruction conversion model, thereby improving the performance and accuracy of the model.

[0124] In some embodiments, step 130 specifically includes:

[0125] Based on the instruction encoder, the structured instructions are segmented and encoded to obtain instruction features;

[0126] Based on a text encoder, the target speech-synthesized text is segmented and encoded to obtain text features;

[0127] Based on the instruction features and the text features, obtain the speech features;

[0128] The synthesized speech is obtained by decoding the speech features.

[0129] The model structure of the instruction encoder is the same as that of the text encoder.

[0130] Optionally, after obtaining the target speech synthesis instruction through step 110 and the structured instruction through step 120, the structured instruction can be input into the instruction encoder, which uses a sub-word unit encoding method to segment and encode instructions of different lengths and complexities to obtain instruction tokens and thus obtain corresponding instruction features; and the target speech synthesis text can be input into the text encoder, which uses a sub-word unit encoding method to segment and encode text of different lengths and complexities to obtain content tokens and thus obtain corresponding text features.

[0131] The instruction encoder and text encoder here are built from the same model structure that integrates word segmentation and encoding, such as the Whisper tokenizer model structure. Because the instruction encoder and text encoder use the same model structure, it ensures that instruction tokens and content tokens reside in the same vector space, facilitating subsequent matching and processing. This unified encoding method improves the model's understanding of instructions and text, enhances the overall system performance, simplifies the system architecture, and reduces the number of parameters and computational complexity.

[0132] Whisper tokenizer is a tokenization and encoding model based on the Transformer architecture. It has powerful text representation capabilities, adopts a multi-layer self-attention mechanism, can capture long-distance dependencies in text, improves encoding accuracy, supports the encoding needs of multiple languages ​​and multiple tasks, and is suitable for unified encoding of instructions and content.

[0133] After obtaining the instruction features and text features, speech feature encoding can be performed based on the instruction features and text features to obtain the corresponding speech features. Then, speech decoding is performed based on the speech features to obtain synthesized speech whose speech content is consistent with the target speech synthesis text and whose speech quality attributes are consistent with the synthesized speech quality attributes indicated by the target speech synthesis instruction.

[0134] The implementation steps for speech features here can include fusing instruction features and text features, then executing pre-configured encoding logic on the fused features or inputting them into a pre-configured encoding model to obtain the corresponding speech features. Feature fusion can be achieved through direct concatenation or by concatenating features according to a set delimiter.

[0135] Here, the steps for implementing synthesized speech can be to input speech features into a speech decoder, so that the speech decoder can perform speech decoding and synthesize synthesized speech that meets the user's expected sound quality attributes.

[0136] The method provided in this embodiment uses an instruction encoder and a text encoder with the same model structure to segment and encode structured instructions and target speech synthesized text, which improves the model's ability to understand instructions and text, simplifies the system architecture, and achieves synthesized speech output with speech content consistent with target text and sound quality attributes that meet the requirements of the instructions.

[0137] In some embodiments, obtaining speech features based on the instruction features and the text features includes:

[0138] Based on a preset delimiter, the instruction features and the text features are concatenated to obtain the target input information;

[0139] The target input information is input into a large language model to obtain the speech features;

[0140] The large-scale language model is used to perform semantic understanding on the target input information, obtain the instruction intent and content information, and synthesize the speech features based on the instruction intent and the content information.

[0141] Optionally, the specific steps for obtaining speech features may further include:

[0142] The instruction features and text features are concatenated according to a preset delimiter to obtain the target input information. For example, special delimiters such as <|promptsep|> can be added between the instruction features and text features to clearly distinguish the two parts, thereby enhancing semantic understanding and better capturing the precise instruction intent and content information.

[0143] In addition, load the parameters and network structure of a pre-trained large language model to ensure that the model is in the correct initial state.

[0144] Subsequently, the spliced ​​target input information is input into a large-scale language model, which internally performs semantic understanding of the target input information through a multi-layer self-attention mechanism, captures the corresponding instruction intent and content information, and encodes and generates corresponding speech features based on the instruction intent and content information.

[0145] The Large Language Model (LLM) mentioned here, often shortened to "large model" or "large language model," refers to a Natural Language Processing (NLP) model with a massive number of parameters. The number of parameters and / or the complexity of the model structure exceed a preset threshold. This model is pre-trained on large-scale text data and possesses high semantic understanding and natural language generation capabilities. The Large Language Model here can be a generative pre-trained Transformer model, etc., but this embodiment does not specifically limit it.

[0146] The method provided in this embodiment concatenates instruction features and text features based on a preset delimiter, inputs them into a large language model for deep semantic understanding, and synthesizes speech features based on the understanding results. This enables the accurate capture of instruction intent and content information, and efficiently generates synthesized speech features that meet expectations, thereby improving the accuracy and naturalness of speech synthesis.

[0147] The following describes the speech synthesis method provided in this embodiment using specific examples.

[0148] Figure 2This is the second flowchart of the speech synthesis method provided by the present invention.

[0149] like Figure 2 As shown, the complete process of the speech synthesis method includes the following steps:

[0150] First, the user's desired synthesized speech quality attributes, i.e., the target speech synthesis instruction, are obtained, along with the content text of the synthesized speech, i.e., the target speech synthesis text.

[0151] Secondly, the free instructions are input into the pre-built instruction conversion model to obtain structured precise instructions, that is, structured instructions;

[0152] Secondly, the precise instructions are input into the instruction encoder to obtain instruction tokens, i.e., instruction features; the target speech-to-text is input into the text encoder to obtain content tokens, i.e., text features.

[0153] Secondly, the instruction tokens and content tokens are concatenated and input into a pre-trained large language model to obtain speech tokens, which are speech features.

[0154] Secondly, the voice tokens are input into the voice decoder to obtain synthesized voice that meets the user's desired sound quality.

[0155] The speech synthesis system provided by the present invention is described below. The speech synthesis system described below can be referred to in correspondence with the speech synthesis method described above.

[0156] Figure 3 This is a schematic diagram of the speech synthesis system provided by the present invention; as shown below. Figure 3 As shown, the system includes: an acquisition unit 310, a processing unit 320, and a synthesis unit 330.

[0157] The acquisition unit 310 is used to acquire the target speech synthesis instruction and the target speech synthesis text;

[0158] The processing unit 320 is used to perform structured processing on the target speech synthesis instruction according to the annotation information of the target speech synthesis instruction to obtain a structured instruction; the annotation information includes word segmentation information, named entity information and phoneme information;

[0159] The synthesis unit 330 is used to perform speech synthesis according to the structured instructions and the target speech synthesis text to obtain synthesized speech.

[0160] The system provided in this embodiment performs structured processing on the target speech synthesis instructions based on the annotation information of the instructions. This enhances the understanding and parsing of implicit information in the instructions by utilizing annotation information such as word segmentation, named entities, and phonemes. Furthermore, it captures potential semantic elements from incomplete instructions, more accurately identifying potential, standardized speech quality attributes within the instructions to obtain structured instructions. Based on these structured instructions and the target speech synthesis text, it controls the generation of synthesized speech, effectively improving the accurate parsing and understanding of implicit information in the instructions. This results in synthesized speech that better meets user expectations, enhancing the accuracy and consistency of the synthesized speech.

[0161] In some embodiments, the processing unit is specifically used for:

[0162] The annotation information of the target speech synthesis instruction is input into the instruction conversion model to obtain the structured instruction;

[0163] The instruction conversion model is obtained by training a pre-trained language model based on sample speech synthesis instructions in different scenarios and the structured labels corresponding to each sample speech synthesis instruction.

[0164] In some embodiments, the system further includes a training unit, specifically used for:

[0165] Based on the annotation information of each sample speech synthesis instruction and the structured label corresponding to each sample speech synthesis instruction, the pre-trained language model is iteratively trained to obtain the first language model;

[0166] If it is found that the first language model fails the performance verification, the structure and parameters of the first language model are adjusted according to the performance verification results of the first language model to obtain the second language model.

[0167] Based on each sample speech synthesis instruction and the sample label corresponding to each sample speech synthesis instruction, the second language model is iteratively trained until a trained second language model that passes performance verification is obtained.

[0168] The instruction conversion model is constructed based on the trained second language model that has passed performance verification;

[0169] The sample labels include structured labels and labels corresponding to language processing tasks, and the language processing tasks include question-answering tasks and / or translation tasks.

[0170] In some embodiments, the training unit is further configured to:

[0171] Each of the sample speech synthesis instructions is subjected to instruction enhancement processing to obtain each enhanced speech synthesis instruction;

[0172] The second language model is iteratively trained based on each of the enhanced speech synthesis instructions and the sample labels corresponding to each of the enhanced speech synthesis instructions;

[0173] The instruction enhancement process includes at least one of synonym replacement, word order reordering, and structural reordering.

[0174] In some embodiments, the training unit is further configured to:

[0175] Each of the sample speech synthesis instructions is subjected to a first preprocessing to obtain preprocessed sample speech synthesis instructions;

[0176] A second preprocessing is performed on each preprocessed sample speech synthesis instruction to obtain the annotation information of each sample speech synthesis instruction;

[0177] The first preprocessing includes data cleaning and / or text planning, and includes word segmentation, named entity recognition, and phoneme annotation.

[0178] In some embodiments, the synthesis unit is specifically used for:

[0179] Based on the instruction encoder, the structured instructions are segmented and encoded to obtain instruction features;

[0180] Based on a text encoder, the target speech-synthesized text is segmented and encoded to obtain text features;

[0181] Based on the instruction features and the text features, obtain the speech features;

[0182] The synthesized speech is obtained by decoding the speech features.

[0183] The model structure of the instruction encoder is the same as that of the text encoder.

[0184] In some embodiments, the synthesis unit is further configured to:

[0185] Based on a preset delimiter, the instruction features and the text features are concatenated to obtain the target input information;

[0186] The target input information is input into a large language model to obtain the speech features;

[0187] The large-scale language model is used to perform semantic understanding on the target input information, obtain the instruction intent and content information, and synthesize the speech features based on the instruction intent and the content information.

[0188] The system provided by this invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0189] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440, wherein the processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a speech synthesis method. This method includes: acquiring a target speech synthesis instruction and a target speech synthesis text; performing structured processing on the target speech synthesis instruction based on annotation information of the target speech synthesis instruction to obtain a structured instruction; the annotation information includes word segmentation information, named entity information, and phoneme information; and performing speech synthesis based on the structured instruction and the target speech synthesis text to obtain synthesized speech.

[0190] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0191] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the speech synthesis method provided by the above methods. The method includes: acquiring a target speech synthesis instruction and a target speech synthesis text; performing structured processing on the target speech synthesis instruction according to the annotation information of the target speech synthesis instruction to obtain a structured instruction; the annotation information includes word segmentation information, named entity information, and phoneme information; and performing speech synthesis according to the structured instruction and the target speech synthesis text to obtain synthesized speech.

[0192] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the speech synthesis method provided by the above methods. The method includes: acquiring a target speech synthesis instruction and a target speech synthesis text; performing structured processing on the target speech synthesis instruction according to the annotation information of the target speech synthesis instruction to obtain a structured instruction; the annotation information includes word segmentation information, named entity information, and phoneme information; and performing speech synthesis according to the structured instruction and the target speech synthesis text to obtain synthesized speech.

[0193] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0194] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech synthesis method, characterized in that, include: Obtain the target speech synthesis instruction and the target speech synthesis text; Based on the annotation information of the target speech synthesis instruction, the target speech synthesis instruction is subjected to structured processing to obtain a structured instruction; the annotation information includes word segmentation information, named entity information, and phoneme information; Speech synthesis is performed based on the structured instructions and the target speech synthesis text to obtain synthesized speech; The step of performing structured processing on the target speech synthesis instruction based on the annotation information of the target speech synthesis instruction to obtain structured instructions includes: The annotation information of the target speech synthesis instruction is input into the instruction conversion model to obtain the structured instruction; The instruction conversion model is obtained by training a pre-trained language model based on sample speech synthesis instructions in different scenarios and the structured labels corresponding to each sample speech synthesis instruction.

2. The speech synthesis method according to claim 1, characterized in that, The training steps of the instruction conversion model include: Based on the annotation information of each sample speech synthesis instruction and the structured label corresponding to each sample speech synthesis instruction, the pre-trained language model is iteratively trained to obtain the first language model; If it is found that the first language model fails the performance verification, the structure and parameters of the first language model are adjusted according to the performance verification results of the first language model to obtain the second language model. Based on each sample speech synthesis instruction and the sample label corresponding to each sample speech synthesis instruction, the second language model is iteratively trained until a trained second language model that passes performance verification is obtained. The instruction conversion model is constructed based on the trained second language model that has passed performance verification; The sample labels include structured labels and labels corresponding to language processing tasks, and the language processing tasks include question-answering tasks and / or translation tasks.

3. The speech synthesis method according to claim 2, characterized in that, The step of iteratively training the second language model based on each sample speech synthesis instruction and the corresponding sample label includes: Each of the sample speech synthesis instructions is subjected to instruction enhancement processing to obtain each enhanced speech synthesis instruction; The second language model is iteratively trained based on each of the enhanced speech synthesis instructions and the sample labels corresponding to each of the enhanced speech synthesis instructions; The instruction enhancement process includes at least one of synonym replacement, word order reordering, and structural reordering.

4. The speech synthesis method according to claim 2, characterized in that, The steps for obtaining the annotation information of each sample speech synthesis instruction include: Each of the sample speech synthesis instructions is subjected to a first preprocessing to obtain preprocessed sample speech synthesis instructions; A second preprocessing is performed on each preprocessed sample speech synthesis instruction to obtain the annotation information of each sample speech synthesis instruction; The first preprocessing includes data cleaning and / or text planning, and includes word segmentation, named entity recognition, and phoneme annotation.

5. The speech synthesis method according to any one of claims 1-4, characterized in that, The step of synthesizing speech based on the structured instructions and the target speech text to obtain synthesized speech includes: Based on the instruction encoder, the structured instructions are segmented and encoded to obtain instruction features; Based on a text encoder, the target speech-synthesized text is segmented and encoded to obtain text features; Based on the instruction features and the text features, obtain the speech features; The synthesized speech is obtained by decoding the speech features. The model structure of the instruction encoder is the same as that of the text encoder.

6. The speech synthesis method according to claim 5, characterized in that, The step of obtaining speech features based on the instruction features and the text features includes: Based on a preset delimiter, the instruction features and the text features are concatenated to obtain the target input information; The target input information is input into a large language model to obtain the speech features; The large-scale language model is used to perform semantic understanding on the target input information, obtain the instruction intent and content information, and synthesize the speech features based on the instruction intent and the content information.

7. A speech synthesis system, characterized in that, include: The acquisition unit is used to acquire the target speech synthesis instruction and the target speech synthesis text; The processing unit is configured to perform structured processing on the target speech synthesis instruction based on the annotation information of the target speech synthesis instruction to obtain a structured instruction; The annotation information includes word segmentation information, named entity information, and phoneme information; A synthesis unit is used to perform speech synthesis based on the structured instructions and the target speech synthesis text to obtain synthesized speech; The processing unit is specifically used for: The annotation information of the target speech synthesis instruction is input into the instruction conversion model to obtain the structured instruction; The instruction conversion model is obtained by training a pre-trained language model based on sample speech synthesis instructions in different scenarios and the structured labels corresponding to each sample speech synthesis instruction.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the speech synthesis method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the speech synthesis method as described in any one of claims 1 to 6.

Images