Speech synthesis method and system, electronic equipment and storage medium

Abstract

The invention provides a speech synthesis method and system, electronic equipment and a storage medium. The method comprises the following steps: acquiring a target speech synthesis instruction and a target speech synthesis text; according to the annotation information of the target speech synthesis instruction, performing structured processing on the target speech synthesis instruction to obtain a structured instruction; the annotation information comprises word segmentation information, named entity information and phoneme information; and performing speech synthesis according to the structured instruction and the target speech synthesis text to obtain synthesized speech. According to the invention, the ability to understand and analyze the implicit information in the free instruction is enhanced to dynamically generate the corresponding speech synthesis parameters, so that the synthesized speech more conforming to the expectation of the user is generated, and the accuracy and consistency of the synthesized speech are effectively enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a speech synthesis method, system, electronic device and storage medium. Background Art

[0002] In the field of natural language processing technology, command-based speech synthesis technology has been widely used because it can flexibly control various attributes of synthesized speech, such as gender, age, timbre, emotion, and speaking speed, and enables users or developers to guide the speech synthesis process through clear instructions to generate speech output that meets specific needs.

[0003] At present, in the process of speech synthesis, a speech synthesis model is usually used to parse structured, predefined standardized instruction information to obtain key synthesis information, and then speech synthesis is performed based on this key synthesis information to generate speech output that meets the instruction requirements.

[0004] In actual applications, user input lacks structured information, the selection and description of command words are not accurate, and key information is often omitted from non-standardized command information. This uncertainty makes it difficult for current speech synthesis models to accurately parse and understand commands, resulting in the generated speech results not meeting user expectations, which in turn affects the accuracy and consistency of the synthesized speech. Summary of the invention

[0005] The present invention provides a speech synthesis method, system, electronic device and storage medium, which are used to solve the defects of poor accuracy and consistency of synthesized speech in the prior art, and improve the accuracy and consistency of synthesized speech by enhancing the model's understanding and parsing ability of implicit information in non-standardized instruction information.

[0006] The present invention provides a speech synthesis method, comprising: Obtaining target speech synthesis instructions and target speech synthesis text; According to the annotation information of the target speech synthesis instruction, the target speech synthesis instruction is structured to obtain a structured instruction; the annotation information includes word segmentation information, named entity information and phoneme information; Speech synthesis is performed according to the structured instruction and the target speech synthesis text to obtain synthesized speech.

[0007] According to the present invention, a speech synthesis method is provided, wherein the target speech synthesis instruction is structured according to the annotation information of the target speech synthesis instruction to obtain a structured instruction, including: Inputting the annotation information of the target speech synthesis instruction into the instruction conversion model to obtain the structured instruction; The instruction conversion model is obtained by training a pre-trained language model based on sample speech synthesis instructions in different scenarios and structured labels corresponding to each of the sample speech synthesis instructions.

[0008] According to the present invention, a speech synthesis method is provided, wherein the training step of the instruction conversion model comprises: Iteratively training the pre-trained language model according to the annotation information of each of the sample speech synthesis instructions and the structured label corresponding to each of the sample speech synthesis instructions to obtain a first language model; When it is learned through verification that the first language model fails the performance verification, adjusting the structure and parameters of the first language model according to the performance verification result of the first language model to obtain a second language model; Iteratively training the second language model according to each of the sample speech synthesis instructions and the sample labels corresponding to each of the sample speech synthesis instructions until a trained second language model that passes performance verification is obtained; constructing the instruction conversion model according to the trained second language model that has passed performance verification; The sample labels include structured labels and labels corresponding to language processing tasks, and the language processing tasks include question-answering tasks and / or translation tasks.

[0009] According to the present invention, a speech synthesis method is provided, wherein the second language model is iteratively trained according to each of the sample speech synthesis instructions and the sample labels corresponding to each of the sample speech synthesis instructions, comprising: Performing instruction enhancement processing on each of the sample speech synthesis instructions respectively to obtain each enhanced speech synthesis instruction; Iteratively training the second language model according to each of the enhanced speech synthesis instructions and the sample labels corresponding to each of the enhanced speech synthesis instructions; The instruction enhancement processing includes at least one of synonym replacement, word order reorganization and structure reorganization.

[0010] According to the present invention, a speech synthesis method is provided, wherein the step of obtaining the annotation information of each sample speech synthesis instruction comprises: Performing a first preprocessing on each of the sample speech synthesis instructions to obtain a preprocessed sample speech synthesis instruction; Performing a second preprocessing on each preprocessed sample speech synthesis instruction to obtain labeling information of each sample speech synthesis instruction; Among them, the first preprocessing includes data cleaning and / or text planning processing, and the first preprocessing includes word segmentation processing, named entity recognition and phoneme tagging.

[0011] According to the present invention, a speech synthesis method is provided, wherein the speech synthesis is performed according to the structured instruction and the target speech synthesis text to obtain the synthesized speech, comprising: Based on the instruction encoder, the structured instruction is segmented and encoded to obtain instruction features; Based on a text encoder, segment and encode the target speech synthesis text to obtain text features; Acquire voice features according to the instruction features and the text features; Performing speech decoding on the speech feature to obtain the synthesized speech; Among them, the model structure of the instruction encoder is the same as the model structure of the text encoder.

[0012] According to the present invention, a speech synthesis method is provided, wherein the acquiring of speech features according to the instruction features and the text features comprises: Based on a preset separator, the instruction feature and the text feature are concatenated to obtain target input information; Inputting the target input information into a large language model to obtain the speech features; The large language model is used to perform semantic understanding on the target input information, obtain instruction intent and content information, and synthesize the speech features according to the instruction intent and the content information.

[0013] The present invention also provides a speech synthesis system, comprising: Obtaining target speech synthesis instructions and target speech synthesis text; According to the annotation information of the target speech synthesis instruction, the target speech synthesis instruction is structured to obtain a structured instruction; the annotation information includes word segmentation information, named entity information and phoneme information; Speech synthesis is performed according to the structured instruction and the target speech synthesis text to obtain synthesized speech.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned speech synthesis methods is implemented.

[0015] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the speech synthesis method described in any one of the above is implemented.

[0016] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned speech synthesis methods.

[0017] The speech synthesis method, system, electronic device and storage medium provided by the present invention perform structured processing on the target speech synthesis instruction according to the annotation information of the target speech synthesis instruction, so as to enhance the understanding and parsing of the implicit information in the target speech synthesis instruction by utilizing the annotation information such as word segmentation, named entity and phoneme in the annotation information, and then capture the potential semantic elements from the incomplete instruction, so as to more accurately identify the potential and standardized speech quality attributes in the target speech synthesis instruction, obtain the structured instruction, and control the generation of the synthesized speech based on the structured instruction and the target speech synthesis text, so as to effectively improve the accurate parsing and understanding ability of the implicit information in the instruction, thereby generating the synthesized speech that better meets the user's expectations and enhancing the accuracy and consistency of the synthesized speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 This is one of the flow charts of the speech synthesis method provided by the present invention.

[0020] Figure 2 This is the second flow chart of the speech synthesis method provided by the present invention.

[0021] Figure 3 It is a structural schematic diagram of the speech synthesis system provided by the present invention.

[0022] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] In the field of speech synthesis, the basic properties of speech synthesis technology rely on information provided by existing data, such as gender, age, timbre, and emotion, etc. This information provides a solid foundation for generating natural and expressive speech.

[0025] In existing speech synthesis frameworks, the synthesis process is usually guided by predefined instructions. These instructions often contain clear and key synthesis parameters, such as speech rate and emotional intensity, which can help the model accurately understand and perform speech synthesis tasks. In this case, the synthesis model can efficiently generate speech that meets expectations, ensuring the naturalness and controllability of the sound. The effectiveness of this process depends on the clarity and completeness of the instruction information.

[0026] Therefore, current command speech synthesis technology usually uses a speech synthesis model to parse structured, predefined standardized command information to obtain key synthesis information such as emotion, timbre, speaking speed and other parameters, and performs speech synthesis based on this key synthesis information to generate speech output that meets the command requirements, that is, to ensure that the synthesized speech can accurately express the speech attributes required by the user.

[0027] This type of speech synthesis method shows high accuracy when processing clear instructions, because the model can generate speech results with specific timbre and emotion based on clear parameter configuration. In terms of technical implementation, this type of speech synthesis method usually uses a preset corpus and a specific labeling system to train the model through supervised learning, so that it can extract key synthesis parameters from the label information in the instruction. This structured approach greatly simplifies the speech synthesis process and improves the stability and consistency of the generated speech.

[0028] However, when speech synthesis is faced with free instructions from users, that is, non-standardized instructions that lack structured information, the selection and description of instruction words are not precise, and key information is often omitted, existing speech synthesis methods often face significant limitations. Specifically, free instructions lack structured information, the selection and description of instruction words are not precise, and key information is often omitted. This uncertainty makes it difficult for existing synthesis models to accurately parse and understand instructions, resulting in the generated speech results not meeting user expectations. For example, the user may only provide a vague description of the emotion without clearly indicating the intensity or details of the emotion, resulting in less than ideal synthesized speech effects. In addition, in the absence of clear labels, the model is prone to deviations when extracting effective information from vague instructions, which affects the accuracy and consistency of the synthesized speech.

[0029] In order to solve the defects of the prior art in the accuracy and consistency of synthesized speech under free instructions, the present application provides a speech synthesis method, which improves the accuracy and consistency of synthesized speech by enhancing the ability to understand and parse implicit information in free instructions to dynamically generate corresponding speech synthesis parameters.

[0030] It should be noted that the execution subject of the method may be a speech synthesis system, which may be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer, a netbook or a personal digital assistant, etc., and the non-mobile electronic device may be a server, a network attached storage, a personal computer, etc., which is not specifically limited in the present invention.

[0031] Figure 1 is one of the flow charts of the speech synthesis method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 and step 130 .

[0032] Step 110, obtaining a target speech synthesis instruction and a target speech synthesis text.

[0033] Here, the target speech synthesis text represents the content text used for synthesizing speech, and the target speech synthesis text may include one or more sentences, such as the text of a paragraph, or the text of an entire chapter containing multiple paragraphs. The target speech synthesis text may be directly input by a user, or may be obtained by acquiring an image through an image acquisition device such as a scanner, a mobile phone, or a camera, and performing optical character recognition on the image, or may be obtained by performing speech transcription on the acquired speech, or may be obtained by crawling the Internet, which is not specifically limited in the embodiments of the present invention.

[0034] The target speech synthesis instruction is a free instruction that describes the speech quality attributes expected when synthesizing speech in natural language; the speech quality attributes include but are not limited to speech speed, emotional intensity, timbre, gender, age, the environment scene where the speech is located, etc., which are not specifically limited in this embodiment. The target speech synthesis instruction can be directly input by the user, or can be obtained by collecting and transcribing the user input, which is not specifically limited in this embodiment.

[0035] It should be noted that the so-called user input may be information input through a command line interface, a graphical interface, touch input, a drop-down selection input, voice input, gesture input, visual input, brain-computer input, etc. This embodiment does not specifically limit this.

[0036] Step 120, according to the annotation information of the target speech synthesis instruction, the target speech synthesis instruction is structured to obtain a structured instruction; the annotation information includes word segmentation information, named entity information and phoneme information.

[0037] Optionally, after acquiring the target speech synthesis instruction, the target speech synthesis instruction may be annotated to obtain the annotated information of the target speech synthesis instruction. For example, the target speech synthesis instruction may be segmented using a natural speech processing tool or model to obtain the segmentation information of the target speech synthesis instruction to help accurately understand the instruction content. In addition, the target speech synthesis instruction may be identified by named entity recognition to obtain the named entity information of the target speech synthesis instruction to help accurately understand the instruction content. In addition, the target speech synthesis instruction may be annotated at the phoneme level to obtain the phoneme sequence of the target speech synthesis instruction, that is, the phoneme information, to accurately understand the phoneme features of the instruction; then, the annotated information of the target speech synthesis instruction is constructed based on the segmentation information, the named entity information, and the phoneme information.

[0038] After obtaining the annotation information of the target speech synthesis instruction, the target speech synthesis instruction can be structured according to the annotation information of the target speech synthesis instruction to obtain a structured instruction. The structured instruction here is a structured and accurate instruction corresponding to the target speech synthesis instruction with clear instruction word description that can reflect the actual speech quality attributes.

[0039] The structured processing here can be achieved through pre-set logic or through a pre-trained model. For example, template matching can be performed based on the annotation information of the target speech synthesis instruction, so as to perform structured processing of the target speech synthesis instruction based on the matched template, thereby obtaining a structured instruction; for another example, the annotation information of the target speech synthesis instruction is input into a pre-trained instruction conversion model, and the instruction conversion model performs structured processing of the target speech synthesis instruction to obtain a structured instruction. The instruction conversion model here can be formed by collecting a large number of free instructions that describe the speech quality attributes expected when synthesizing speech in natural language, and matching the sample speech synthesis instructions with corresponding structured and precise instructions to form a structured label corresponding to the sample speech synthesis instruction, and then performing model training based on the sample speech synthesis instruction and the structured label corresponding to the sample speech synthesis instruction, so that the trained instruction conversion model can output corresponding structured and precise instructions for the free instructions.

[0040] Step 130, performing speech synthesis according to the structured instruction and the target speech synthesis text to obtain synthesized speech.

[0041] Optionally, after obtaining the structured instruction, the structured instruction and the target speech synthesis text can be combined for speech synthesis to obtain the corresponding synthesized speech. It should be understood that the synthesized speech here is a speech whose speech content is consistent with the target speech synthesis text and whose speech quality attributes are consistent with the synthesized speech quality attributes indicated by the target speech synthesis instruction.

[0042] Here, speech synthesis can be achieved through a pre-trained model; for example, the structured instructions and the target speech synthesis text can be directly input into the pre-trained language model for speech synthesis to output the corresponding synthesized speech, or the structured instructions and the target speech synthesis text can be feature encoded and feature fused respectively, and then the fused features can be input into the pre-trained language model for speech synthesis to output the corresponding synthesized speech.

[0043] It should be noted that no matter whether the target speech synthesis instruction contains one or multiple speech quality attributes, the synthesis model can accurately identify the speech quality attributes and, based on this, realize speech synthesis for speech quality attributes of one or more controllable dimensions, so as to efficiently generate expected speech and ensure the naturalness, controllability and flexibility of the sound.

[0044] The method provided in this embodiment performs structured processing on the target speech synthesis instruction according to the annotation information of the target speech synthesis instruction, so as to enhance the understanding and parsing of the implicit information in the target speech synthesis instruction by utilizing the annotation information such as word segmentation, named entity and phoneme in the annotation information, and then capture the potential semantic elements from the incomplete instruction, so as to more accurately identify the potential and standardized speech quality attributes in the target speech synthesis instruction, obtain the structured instruction, and control the generation of the synthesized speech based on the structured instruction and the target speech synthesis text, so as to effectively improve the ability to accurately parse and understand the implicit information in the instruction, thereby generating a synthesized speech that better meets the user's expectations and enhancing the accuracy and consistency of the synthesized speech.

[0045] In some embodiments, step 120 specifically includes: Inputting the annotation information of the target speech synthesis instruction into the instruction conversion model to obtain the structured instruction; The instruction conversion model is obtained by training a pre-trained language model based on sample speech synthesis instructions in different scenarios and structured labels corresponding to each of the sample speech synthesis instructions.

[0046] Optionally, the structured instructions can be structured instructions obtained by structured processing by an instruction conversion model, with clear instruction word descriptions that can reflect the actual speech quality attributes. That is, the annotation information of the target speech synthesis instructions can be input into the instruction conversion model, and the instruction conversion model can apply the annotation information to perform structured processing on the target speech synthesis instructions to output structured instructions of the target speech synthesis instructions.

[0047] The training steps of the command conversion model here include: collecting a large number of free commands expressed by users in natural voice in different scenarios to form sample speech synthesis commands; and matching corresponding structured and precise commands for each sample speech synthesis command, and expressing them in a specific format or coding language, etc., to ensure that each generated sample speech synthesis command has a corresponding structured label; then, preprocessing the sample speech synthesis commands in different scenarios to obtain the annotation information of each sample speech synthesis command. Using the annotation information of each sample speech synthesis command as a sample and the structured label as a label, the pre-trained language model is trained in a single task or multiple tasks, thereby obtaining instructions that can effectively output corresponding structured commands with precise speech quality attribute indications for free commands, so as to improve the flexibility and accuracy of the speech synthesis system.

[0048] The scenarios include but are not limited to daily life scenarios, office scenarios, entertainment scenarios, and smart home control scenarios, which are not specifically limited in this embodiment.

[0049] The pre-trained language model can be constructed by various pre-trained language models suitable for instruction conversion tasks, such as by constructing a model of a fine-tuned language network with T5 (FLAN-T5) based on the T5 architecture, etc. This embodiment does not specifically limit this. FLAN-T5 is a T5 model that has been fine-tuned by instructions, has powerful natural language understanding and generation capabilities, and can be applied to instruction conversion tasks.

[0050] The method provided in this embodiment, by introducing a more adaptive natural language understanding mechanism in the pre-trained language model, enables the command conversion model to capture potential semantic elements from incomplete command information and dynamically generate corresponding speech synthesis parameters. At the same time, a more intelligent contextual reasoning mechanism in the pre-trained language model will also be introduced to help the command conversion model more accurately identify the user's emotional intention and timbre preference in the absence of clear parameter labels, so as to improve the quality of synthesized speech in free command scenarios, thereby achieving better satisfaction of users' complex and diverse needs.

[0051] In some embodiments, the training step of the instruction conversion model includes: Iteratively training the pre-trained language model according to the annotation information of each of the sample speech synthesis instructions and the structured label corresponding to each of the sample speech synthesis instructions to obtain a first language model; When it is learned through verification that the first language model fails the performance verification, adjusting the structure and parameters of the first language model according to the performance verification result of the first language model to obtain a second language model; Iteratively training the second language model according to each of the sample speech synthesis instructions and the sample labels corresponding to each of the sample speech synthesis instructions until a trained second language model that passes performance verification is obtained; constructing the instruction conversion model according to the trained second language model that has passed performance verification; The sample labels include structured labels and labels corresponding to language processing tasks, and the language processing tasks include question-answering tasks and / or translation tasks.

[0052] Optionally, the training steps of the instruction conversion model specifically include: a model design step and a model fine-tuning step, a model verification step and a model optimization step.

[0053] The model design steps include the following steps: Step S10, selecting a model architecture of a pre-trained language model, such as selecting a FLAN-T5 model structure as the model architecture of the pre-trained language model; Step S11, selecting an appropriate model size according to computing resources and task requirements, such as selecting a small T5 model (referred to as T5-small), a medium-sized T5 model (referred to as T5-base), or a large-scale T5 model (referred to as T5-large); In step S12, model parameters of the model architecture of the selected model size are used as initial values ​​to construct a pre-trained language model.

[0054] The model fine-tuning step includes the following steps: Step S20, input and output format definition: determine the input and output formats of the model, take the annotation information of the sample speech synthesis instructions (i.e., sample free instructions) as the input sequence, and the structured label as the target output sequence; Step S21, loss function setting: constructing a loss function according to the difference between the model output and the structured label, such as using a cross entropy loss function or other loss functions, which is not specifically limited in this embodiment; Step S22, fine-tuning training: perform multiple rounds of iterative training on the pre-trained language model according to the annotation information of each sample speech synthesis instruction in the training set and the structured label corresponding to each sample speech synthesis instruction, traversing all the training data in each round, and evaluating the model performance on the validation set after each training cycle, monitoring the changes in the loss function and the evaluation index until the maximum number of iterations is reached, or the loss function converges or the evaluation index converges, then stop the training, and obtain the pre-trained language model with the best comprehensive loss function and evaluation index in the training process to construct the first language model.

[0055] The model validation steps specifically include: The performance indicators are calculated using the annotation information of each sample speech synthesis instruction in the test set and the structured labels corresponding to each sample speech synthesis instruction to verify and measure the instruction conversion performance of the first language model. The performance indicators include but are not limited to Bilingual Evaluation Understudy (BLEU), Recall-Oriented Understudy for Gisting Evaluation (ROUGE), accuracy, etc.

[0056] When it is learned through verification that the first language model passes the verification, such as when the values ​​corresponding to each performance indicator satisfy the corresponding indicator threshold, or when the fusion indicator values ​​corresponding to multiple performance indicators satisfy the corresponding fusion indicator threshold, the instruction conversion model is directly constructed based on the first language model.

[0057] If it is found through verification that the first language model has not passed the verification, such as when the value corresponding to any performance indicator does not meet its corresponding indicator threshold, or when the fusion indicator values ​​corresponding to multiple performance indicators do not meet the corresponding fusion indicator threshold, the error conversion cases of the first language model can be analyzed to understand the causes of the errors, such as comprehension errors, generation errors, etc.; or, based on manual evaluation of the structured instructions output by the model by professionals, their correctness and usability can be examined to analyze the reasons why the first language model has not passed the verification, so that model tuning training can be performed later on the basis of the first language model.

[0058] When it is found that the first language model fails the performance verification, the following model optimization steps are performed: According to the performance verification results of the first language model, the structure and parameters of the first language model are adjusted to obtain the second language model; and according to the sample labels of the multi-tasks corresponding to the sample speech synthesis instructions in the training set and the sample speech synthesis instructions, the second language model is optimized and trained, and the above model verification steps and model optimization steps are repeated until the trained second language model that passes the performance verification is obtained, and then the training is stopped to improve the instruction conversion performance. The trained second language model that passes the performance verification is saved for subsequent instruction conversion task processing, and the final instruction conversion model can be deployed and constructed based on the trained second language model that passes the performance verification for instruction structuring processing.

[0059] Here, the optimization training of the second language model can be to directly train the second language model using each sample speech synthesis instruction and the sample labels of the multiple tasks corresponding to each sample speech synthesis instruction, or to first pre-process each sample speech synthesis instruction, such as after data enhancement, and then train the second language model based on the pre-processed sample speech synthesis instructions and the sample labels of the multiple tasks corresponding to the pre-processed sample speech synthesis instructions, etc. This embodiment does not specifically limit this.

[0060] Here, the sample labels of multiple tasks include but are not limited to the labels corresponding to the instruction conversion task, that is, the structured labels, as well as the labels corresponding to the question and answer task and / or the labels corresponding to the translation task. By jointly training the instruction conversion task with related natural language processing tasks (such as question and answer, translation), the generalization ability of the model can be effectively improved.

[0061] The method provided in this embodiment first forms a preliminary first language model by iterative training based on a pre-trained language model; after verifying its performance, fine-tuning the structure and parameters of the non-compliant parts is performed to obtain a second language model; and then joint training is performed based on sample labels of multiple tasks, including tasks such as instruction conversion, question answering, and translation, which significantly enhances the generalization ability of the model. Finally, the trained second language model that has passed performance verification is used to construct an instruction conversion model, which realizes precise structured processing of sample speech synthesis instructions, and effectively improves the accuracy and efficiency of instruction conversion.

[0062] In some embodiments, the iterative training of the second language model according to each of the sample speech synthesis instructions and the sample label corresponding to each of the sample speech synthesis instructions includes: Performing instruction enhancement processing on each of the sample speech synthesis instructions respectively to obtain each enhanced speech synthesis instruction; Iteratively training the second language model according to each of the enhanced speech synthesis instructions and the sample labels corresponding to each of the enhanced speech synthesis instructions; The instruction enhancement processing includes at least one of synonym replacement, word order reorganization and structure reorganization.

[0063] Optionally, the model optimization step further includes: Perform synonym replacement on each sample speech synthesis instruction (i.e., sample free instruction) to increase expression diversity, and / or perform instruction reorganization on each sample speech synthesis instruction, i.e., change the word order or structure of the instruction to generate new training samples, thereby obtaining enhanced speech synthesis instructions; then, iteratively train the second language model according to the enhanced speech synthesis instruction and the sample label corresponding to the enhanced speech synthesis instruction, and repeat the above-mentioned model verification step and model optimization step until a trained second language model that passes performance verification is obtained, then stop training to improve instruction conversion performance. The trained second language model that passes performance verification is saved for subsequent instruction conversion task processing, and the final instruction conversion model can be constructed based on the trained second language model that passes performance verification for instruction structuring processing.

[0064] The method provided in this embodiment increases the diversity and coverage of training data by performing instruction enhancement processing on sample speech synthesis instructions, such as synonym replacement, word order reorganization and structure reorganization, and performs model training based on this, which not only improves the model's adaptability to different expressions, but also enhances the model's understanding of the potential semantics in the instructions, so that the model can more accurately convert unstructured speech synthesis instructions into structured instructions, thereby improving the accuracy and consistency of synthesized speech and user satisfaction in practical applications.

[0065] In some embodiments, the step of acquiring the annotation information of each of the sample speech synthesis instructions includes: Performing a first preprocessing on each of the sample speech synthesis instructions to obtain a preprocessed sample speech synthesis instruction; Performing a second preprocessing on each preprocessed sample speech synthesis instruction to obtain labeling information of each sample speech synthesis instruction; Among them, the first preprocessing includes data cleaning and / or text planning processing, and the first preprocessing includes word segmentation processing, named entity recognition and phoneme tagging.

[0066] Optionally, the step of acquiring the annotation information of each of the sample speech synthesis instructions specifically includes: Step S00, collecting a large number of free instructions expressed by users in natural language to form sample speech synthesis instructions; Step S01, preprocessing the collected sample speech synthesis instructions, such as one or more of noise reduction, removal of silent segments and normalization processing, and information annotation processing, to improve data quality. The specific implementation steps include: Step S011, performing data cleaning processing on the sample speech synthesis instructions, specifically including removing repeated, erroneous or incomplete instruction pairs to correct spelling and grammatical errors and ensure data quality; Step S012, performing text normalization processing on the sample speech synthesis instruction, specifically including performing unified text normalization processing on the sample speech synthesis instruction and structured tags, such as lowercase conversion, punctuation processing, special character escape, etc.; Step S013, segmenting and annotating the preprocessed sample speech synthesis instruction, specifically including segmenting the preprocessed sample speech synthesis instruction, and performing part-of-speech tagging or named entity recognition as needed to assist the model in understanding the instruction content; Step S014, performing phoneme annotation on the preprocessed sample speech synthesis instruction, specifically including using a professional annotation tool to perform phoneme-level annotation on the preprocessed speech data to generate a corresponding phoneme sequence, that is, phoneme information; By processing each sample speech synthesis instruction through the above steps, the labeling information of each sample speech synthesis instruction can be obtained. Subsequently, based on the labeling information of each sample speech synthesis instruction, the model is iterated and trained to quickly and accurately obtain an instruction conversion model that can accurately output structured instructions based on free instructions.

[0067] The method provided in this embodiment can efficiently and accurately extract high-quality annotation information from sample free instructions. This information includes the processing results of data cleaning, text normalization, word segmentation, named entity recognition and phoneme tagging, which provides rich and accurate training data for building an instruction conversion model with excellent performance, thereby improving the performance and accuracy of the model.

[0068] In some embodiments, step 130 specifically includes: Based on the instruction encoder, the structured instruction is segmented and encoded to obtain instruction features; Based on a text encoder, segment and encode the target speech synthesis text to obtain text features; Acquire voice features according to the instruction features and the text features; Performing speech decoding on the speech feature to obtain the synthesized speech; Among them, the model structure of the instruction encoder is the same as the model structure of the text encoder.

[0069] Optionally, after obtaining the target speech synthesis instruction through step 110 and obtaining the structured instruction through step 120, the structured instruction can be input into the instruction encoder, so that the instruction encoder can implement word segmentation and encoding of instructions of different lengths and complexities based on the encoding method of sub-word units to obtain instruction Tokens, and then obtain corresponding instruction features; and the target speech synthesis text can be input into the text encoder, so that the text encoder can implement word segmentation and encoding of texts of different lengths and complexities based on the encoding method of sub-word units to obtain content Tokens, and then obtain corresponding text features.

[0070] The instruction encoder and text encoder here are constructed by the same model structure with word segmentation and encoding, such as both are constructed based on the Whisper tokenizer model structure. Since the instruction encoder and the text encoder use the same model structure, it ensures that the instruction tokens and content tokens are in the same vector space, which is convenient for subsequent matching and processing. Therefore, through a unified encoding method, the model's ability to understand instructions and text is improved, the overall performance of the system is enhanced, the system architecture is simplified, and the number of parameters and computational complexity are reduced.

[0071] Among them, Whisper tokenizer is a word segmentation and encoding model based on the Transformer architecture. It has powerful text representation capabilities and adopts a multi-layer self-attention mechanism. It can capture long-distance dependencies in the text, improve the accuracy of encoding, support multi-language and multi-task encoding requirements, and is suitable for unified encoding of instructions and content.

[0072] After obtaining the instruction features and text features, speech feature encoding can be performed based on the instruction features and text features to obtain corresponding speech features. Speech decoding can be performed based on the speech features to obtain synthesized speech whose speech content is consistent with the target speech synthesis text and whose speech quality attributes are consistent with the synthesized speech quality attributes indicated by the target speech synthesis instruction.

[0073] The implementation step of the speech feature here can be to fuse the instruction feature and the text feature, so as to execute the pre-configured encoding logic on the fused feature or input it into the pre-configured encoding model to obtain the corresponding speech feature. The feature fusion can be realized by direct concatenation or splicing according to the set delimiter.

[0074] Here, the implementation step of synthesizing speech may be to input speech features into a speech decoder so that the speech decoder performs speech decoding to synthesize a synthesized speech that meets the user's expected sound quality attributes.

[0075] The method provided in this embodiment utilizes an instruction encoder and a text encoder with the same model structure to segment and encode structured instructions and target speech synthesis text, thereby improving the model's ability to understand instructions and text, simplifying the system architecture, and achieving synthesized speech output whose speech content is consistent with the target text and whose sound quality attributes meet the instruction requirements.

[0076] In some embodiments, acquiring the voice feature according to the instruction feature and the text feature includes: Based on a preset separator, the instruction feature and the text feature are concatenated to obtain target input information; Inputting the target input information into a large language model to obtain the speech features; The large language model is used to perform semantic understanding on the target input information, obtain instruction intent and content information, and synthesize the speech features according to the instruction intent and the content information.

[0077] Optionally, the specific step of acquiring the voice feature further includes: According to the preset separator, the command features and text features are concatenated to obtain the target input information. For example, special separator tokens such as <|promptsep|> are added between the command features and the text features to clearly distinguish the two parts, thereby enhancing semantic understanding and better capturing the precise command intent and content information.

[0078] In addition, the parameters and network structure of the pre-trained large language model are loaded to ensure that the model is in the correct initial state.

[0079] Then, the concatenated target input information is input into a large language model, which uses a multi-layer self-attention mechanism to semantically understand the target input information, capture the corresponding instruction intent and content information, and encode the instruction intent and content information to generate corresponding speech features.

[0080] The large language model (LLM) referred to here as a large model or a large language model refers to a natural language processing (NLP) model with a large parameter scale. The number of model parameters and / or the complexity of the model structure exceeds a preset threshold. The model is pre-trained with large-scale text data and has a high degree of semantic understanding and the ability to generate natural language. The large language model here can be a generative pre-trained Transformer model, etc., which is not specifically limited in this embodiment.

[0081] The method provided in this embodiment concatenates instruction features and text features based on preset delimiters, inputs them into a large language model for deep semantic understanding, and synthesizes speech features based on the understanding results, so as to accurately capture the instruction intent and content information, and efficiently generate expected synthesized speech features, thereby improving the accuracy and naturalness of speech synthesis.

[0082] The speech synthesis method provided in this embodiment is described below with reference to a specific example.

[0083] Figure 2 This is the second flow chart of the speech synthesis method provided by the present invention.

[0084] like Figure 2 As shown in FIG. 1 , the complete process of the speech synthesis method specifically includes the following steps: First, the user's free instructions for the desired sound quality attributes of the synthesized speech, that is, the target speech synthesis instructions, and the content text of the synthesized speech, that is, the target speech synthesis text, are obtained respectively.

[0085] Secondly, the free instructions are input into the pre-built instruction conversion model to obtain structured precise instructions, namely structured instructions; Secondly, the precise instruction is input into the instruction encoder to obtain instruction tokens, i.e. instruction features; the target speech synthesis text is input into the text encoder to obtain content tokens, i.e. text features; Secondly, after concatenating the command tokens and content tokens, they are input into the pre-trained large language model to obtain speech tokens, i.e. speech features. Secondly, the speech tokens are input into the speech decoder to obtain the synthesized speech that meets the user's expected sound quality.

[0086] The speech synthesis system provided by the present invention is described below. The speech synthesis system described below and the speech synthesis method described above can be referred to each other.

[0087] Figure 3 Schematic diagram of the structure of the speech synthesis system provided by the present invention; Figure 3 As shown, the system includes: an acquisition unit 310, a processing unit 320 and a synthesis unit 330.

[0088] The acquisition unit 310 is used to acquire a target speech synthesis instruction and a target speech synthesis text; The processing unit 320 is used to perform structured processing on the target speech synthesis instruction according to the annotation information of the target speech synthesis instruction to obtain a structured instruction; the annotation information includes word segmentation information, named entity information and phoneme information; The synthesis unit 330 is used to perform speech synthesis according to the structured instruction and the target speech synthesis text to obtain synthesized speech.

[0089] The system provided in this embodiment performs structured processing on the target speech synthesis instruction according to the annotation information of the target speech synthesis instruction, so as to enhance the understanding and parsing of the implicit information in the target speech synthesis instruction by utilizing the annotation information such as word segmentation, named entity and phoneme in the annotation information, and then capture the potential semantic elements from the incomplete instruction, so as to more accurately identify the potential and standardized speech quality attributes in the target speech synthesis instruction, obtain the structured instruction, and control the generation of the synthesized speech based on the structured instruction and the target speech synthesis text, so as to effectively improve the ability to accurately parse and understand the implicit information in the instruction, thereby generating a synthesized speech that better meets the user's expectations and enhancing the accuracy and consistency of the synthesized speech.

[0090] In some embodiments, the processing unit is specifically configured to: Inputting the annotation information of the target speech synthesis instruction into the instruction conversion model to obtain the structured instruction; The instruction conversion model is obtained by training a pre-trained language model based on sample speech synthesis instructions in different scenarios and structured labels corresponding to each of the sample speech synthesis instructions.

[0091] In some embodiments, the system further comprises a training unit, specifically configured to: Iteratively training the pre-trained language model according to the annotation information of each of the sample speech synthesis instructions and the structured label corresponding to each of the sample speech synthesis instructions to obtain a first language model; When it is learned through verification that the first language model fails the performance verification, adjusting the structure and parameters of the first language model according to the performance verification result of the first language model to obtain a second language model; Iteratively training the second language model according to each of the sample speech synthesis instructions and the sample labels corresponding to each of the sample speech synthesis instructions until a trained second language model that passes performance verification is obtained; constructing the instruction conversion model according to the trained second language model that has passed performance verification; The sample labels include structured labels and labels corresponding to language processing tasks, and the language processing tasks include question-answering tasks and / or translation tasks.

[0092] In some embodiments, the training unit is further configured to: Performing instruction enhancement processing on each of the sample speech synthesis instructions respectively to obtain each enhanced speech synthesis instruction; Iteratively training the second language model according to each of the enhanced speech synthesis instructions and the sample labels corresponding to each of the enhanced speech synthesis instructions; The instruction enhancement processing includes at least one of synonym replacement, word order reorganization and structure reorganization.

[0093] In some embodiments, the training unit is further configured to: Performing a first preprocessing on each of the sample speech synthesis instructions to obtain a preprocessed sample speech synthesis instruction; Performing a second preprocessing on each preprocessed sample speech synthesis instruction to obtain labeling information of each sample speech synthesis instruction; Among them, the first preprocessing includes data cleaning and / or text planning processing, and the first preprocessing includes word segmentation processing, named entity recognition and phoneme tagging.

[0094] In some embodiments, the synthesis unit is specifically used for: Based on the instruction encoder, the structured instruction is segmented and encoded to obtain instruction features; Based on a text encoder, segment and encode the target speech synthesis text to obtain text features; Acquire voice features according to the instruction features and the text features; Performing speech decoding on the speech feature to obtain the synthesized speech; Among them, the model structure of the instruction encoder is the same as the model structure of the text encoder.

[0095] In some embodiments, the synthesis unit is further used to: Based on a preset separator, the instruction feature and the text feature are concatenated to obtain target input information; Inputting the target input information into a large language model to obtain the speech features; The large language model is used to perform semantic understanding on the target input information, obtain instruction intent and content information, and synthesize the speech features according to the instruction intent and the content information.

[0096] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0097] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the speech synthesis method, which includes: obtaining a target speech synthesis instruction and a target speech synthesis text; according to the annotation information of the target speech synthesis instruction, performing structured processing on the target speech synthesis instruction to obtain a structured instruction; the annotation information includes word segmentation information, named entity information and phoneme information; performing speech synthesis according to the structured instruction and the target speech synthesis text to obtain a synthesized speech.

[0098] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0099] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech synthesis method provided by the above-mentioned methods, and the method includes: obtaining a target speech synthesis instruction and a target speech synthesis text; according to the annotation information of the target speech synthesis instruction, structuring the target speech synthesis instruction to obtain a structured instruction; the annotation information includes word segmentation information, named entity information and phoneme information; and performing speech synthesis according to the structured instruction and the target speech synthesis text to obtain synthesized speech.

[0100] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the speech synthesis method provided by the above-mentioned methods, the method comprising: obtaining a target speech synthesis instruction and a target speech synthesis text; according to the annotation information of the target speech synthesis instruction, performing structured processing on the target speech synthesis instruction to obtain a structured instruction; the annotation information includes word segmentation information, named entity information and phoneme information; and performing speech synthesis according to the structured instruction and the target speech synthesis text to obtain synthesized speech.

[0101] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0102] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech synthesis method, characterized in that: include: Obtaining target speech synthesis instructions and target speech synthesis text; According to the annotation information of the target speech synthesis instruction, the target speech synthesis instruction is structured to obtain a structured instruction; the annotation information includes word segmentation information, named entity information and phoneme information; Speech synthesis is performed according to the structured instruction and the target speech synthesis text to obtain synthesized speech.

2. The speech synthesis method according to claim 1, characterized in that: The step of performing structured processing on the target speech synthesis instruction according to the annotation information of the target speech synthesis instruction to obtain a structured instruction includes: Inputting the annotation information of the target speech synthesis instruction into the instruction conversion model to obtain the structured instruction; The instruction conversion model is obtained by training a pre-trained language model based on sample speech synthesis instructions in different scenarios and structured labels corresponding to each of the sample speech synthesis instructions.

3. The speech synthesis method according to claim 2, characterized in that: The training steps of the instruction conversion model include: Iteratively training the pre-trained language model according to the annotation information of each of the sample speech synthesis instructions and the structured label corresponding to each of the sample speech synthesis instructions to obtain a first language model; When it is learned through verification that the first language model fails the performance verification, adjusting the structure and parameters of the first language model according to the performance verification result of the first language model to obtain a second language model; Iteratively training the second language model according to each of the sample speech synthesis instructions and the sample labels corresponding to each of the sample speech synthesis instructions until a trained second language model that passes performance verification is obtained; constructing the instruction conversion model according to the trained second language model that has passed performance verification; The sample labels include structured labels and labels corresponding to language processing tasks, and the language processing tasks include question-answering tasks and / or translation tasks.

4. The speech synthesis method according to claim 3, characterized in that: The iterative training of the second language model according to each of the sample speech synthesis instructions and the sample labels corresponding to each of the sample speech synthesis instructions includes: Performing instruction enhancement processing on each of the sample speech synthesis instructions respectively to obtain each enhanced speech synthesis instruction; Iteratively training the second language model according to each of the enhanced speech synthesis instructions and the sample labels corresponding to each of the enhanced speech synthesis instructions; The instruction enhancement processing includes at least one of synonym replacement, word order reorganization and structure reorganization.

5. The speech synthesis method according to claim 3, characterized in that: The step of obtaining the annotation information of each of the sample speech synthesis instructions includes: Performing a first preprocessing on each of the sample speech synthesis instructions to obtain a preprocessed sample speech synthesis instruction; Performing a second preprocessing on each preprocessed sample speech synthesis instruction to obtain labeling information of each sample speech synthesis instruction; Among them, the first preprocessing includes data cleaning and / or text planning processing, and the first preprocessing includes word segmentation processing, named entity recognition and phoneme tagging.

6. The speech synthesis method according to any one of claims 1 to 5, characterized in that: The performing speech synthesis according to the structured instruction and the target speech synthesis text to obtain synthesized speech includes: Based on the instruction encoder, the structured instruction is segmented and encoded to obtain instruction features; Based on a text encoder, segment and encode the target speech synthesis text to obtain text features; Acquire voice features according to the instruction features and the text features; Performing speech decoding on the speech feature to obtain the synthesized speech; Among them, the model structure of the instruction encoder is the same as the model structure of the text encoder.

7. The speech synthesis method according to claim 6, characterized in that: The acquiring of speech features according to the instruction features and the text features includes: Based on a preset separator, the instruction feature and the text feature are concatenated to obtain target input information; Inputting the target input information into a large language model to obtain the speech features; The large language model is used to perform semantic understanding on the target input information, obtain instruction intent and content information, and synthesize the speech features according to the instruction intent and the content information.

8. A speech synthesis system, characterized in that: include: An acquisition unit, used for acquiring a target speech synthesis instruction and a target speech synthesis text; A processing unit, configured to perform structured processing on the target speech synthesis instruction according to the annotation information of the target speech synthesis instruction to obtain a structured instruction; The annotation information includes word segmentation information, named entity information and phoneme information; The synthesis unit is used to perform speech synthesis according to the structured instruction and the target speech synthesis text to obtain synthesized speech.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the speech synthesis method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 7 is implemented.

Images