Speech synthesis method and device, electronic equipment and storage medium

Abstract

The application provides a speech synthesis method and device, electronic equipment and storage medium, belonging to the technical field of artificial intelligence, comprising: obtaining a text to be synthesized and a tone description text for describing non-semantic information of a target speech signal to be synthesized, jointly encoding the text to be synthesized and the tone description text to obtain a mixed word sequence, inputting the mixed word sequence into a tone control synthesis model to obtain an audio word sequence output by the tone control synthesis model, and decoding the audio word sequence to obtain the target speech signal. The speech synthesis method and device, electronic equipment and storage medium provided by the application use the tone description text in natural language form as an additional input parameter, so that the model can directly understand and accurately control the non-semantic attributes of the speech, solve the technical problems of the prior art, such as dependence on fixed labels, coarse control granularity and single expression capability, and significantly improve the controllability, diversity and humanization of the synthesized speech.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a speech synthesis method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the widespread application of large-scale pre-trained models in the field of speech synthesis, the naturalness and human-likeness of synthesized speech have been significantly improved. However, in open application scenarios such as intelligent assistants, in-vehicle systems, audio reading, and short video dubbing, users not only pursue natural and fluent synthesized speech, but also place increasingly higher demands on the controllability and diversity of the emotion, style, and tone of synthesized speech, hoping to automatically generate speech with rich expressiveness.

[0003] In existing technologies, a common approach to control the attributes of synthesized speech is to train a model using a pre-recorded voice library with specific emotion or style tags. During the synthesis phase, different tag IDs are used to invoke the corresponding pronunciation style. Other approaches attempt to achieve richer expressions by constructing descriptive templates or hierarchical tagging systems. For example, by combining attributes such as emotion, intensity, and prosody, structured control commands are generated to guide the speech synthesis process.

[0004] However, the research revealed at least the following shortcomings in the existing technical solutions: First, regardless of whether a fixed label ID or a structured description template is used, the control method is relatively rigid, making it difficult to support fine-grained description of non-semantic information of speech using free and flexible natural language. Second, existing solutions typically model different speech attributes independently or use fixed combination rules. When performing multi-dimensional attribute fusion, there is a lack of effective modeling of the synergistic relationships between attributes, which may result in inconsistent or unnatural performance in the synthesized speech. Summary of the Invention

[0005] This invention provides a speech synthesis method, apparatus, electronic device, and storage medium to overcome the shortcomings of low classification efficiency, long time consumption, and uncontrollable classification accuracy in the prior art, thereby achieving high-efficiency and high-precision speech synthesis.

[0006] This invention provides a speech synthesis method, comprising the following steps:

[0007] Obtain the text to be synthesized and the tone description text, wherein the tone description text is used to describe the non-semantic information of the target speech signal to be synthesized;

[0008] The text to be synthesized and the tone description text are jointly encoded to obtain a mixed word sequence;

[0009] The mixed word sequence is input into the mood-controlled synthesis model to obtain the audio word sequence output by the mood-controlled synthesis model;

[0010] The audio word sequence is decoded to obtain the target speech signal.

[0011] According to a speech synthesis method provided by the present invention, after acquiring the target speech signal, the method further includes performing reinforcement learning and post-training on the tone control synthesis model;

[0012] The post-reinforcement learning training includes iteratively performing the following steps:

[0013] The target speech signal is input into the tone annotation model, and the tone-understanding text output by the tone annotation model is obtained.

[0014] A reward signal is determined based on the semantic similarity between the tone-understanding text and the tone-description text.

[0015] Based on the reward signal, the model parameters of the tone control synthesis model are optimized.

[0016] According to a speech synthesis method provided by the present invention, determining a reward signal based on the semantic similarity between the tone-understanding text and the tone-description text includes:

[0017] The tone understanding text and the tone description text are respectively input into the text encoding model to obtain the first hidden layer semantic vector and the second hidden layer semantic vector output by the text encoding model.

[0018] The cosine similarity between the first hidden layer semantic vector and the second hidden layer semantic vector is used as the reward signal.

[0019] According to a speech synthesis method provided by the present invention, the tone-controlled synthesis model is trained based on the following method:

[0020] Obtain a first training dataset, which includes at least a first training sample, a second training sample, and a third training sample;

[0021] Each of the first training samples includes a first tone description text sample, a first text sample to be synthesized, and a first speech tag corresponding to the first text sample to be synthesized; each of the second training samples includes a second text sample to be synthesized and a second speech tag corresponding to the second text sample to be synthesized; each of the third training samples is a plain text sample;

[0022] The speech-text bimodal language model is trained synchronously using the first training dataset to obtain the tone-controlled synthesis model.

[0023] According to a speech synthesis method provided by the present invention, the step of simultaneously training a speech-text bimodal language model using a first training dataset to obtain the tone-controlled synthesis model includes:

[0024] When the first tone description text sample, the first text sample to be synthesized, and the first preceding audio lexical sequence in the audio lexical sequence corresponding to the first speech tag are used as input to the speech-text bimodal language model, the next audio predicted lexical of the first preceding audio lexical sequence output by the speech-text bimodal language model is obtained; a first loss is calculated based on the difference between the next audio predicted lexical and the next actual audio lexical of the preceding audio lexical sequence, so as to optimize the model parameters of the speech-text bimodal language model according to the first loss;

[0025] When the second preceding audio lexical subsequence in the audio lexical sequence corresponding to the second speech tag and the second text sample to be synthesized is used as the input of the speech-text bimodal language model, the next audio predicted lexical of the second preceding audio lexical subsequence output by the speech-text bimodal language model is obtained; a second loss is calculated based on the difference between the next audio predicted lexical and the next actual audio lexical of the second preceding audio lexical subsequence, so as to optimize the model parameters of the speech-text bimodal language model according to the second loss;

[0026] When the third preceding text word subsequence in the plain text sample is used as input to the speech-text bimodal language model, the next text predicted word of the third preceding text word subsequence output by the speech-text bimodal language model is obtained; a third loss is calculated based on the difference between the next text predicted word and the next text true word of the third preceding text word subsequence, so as to optimize the model parameters of the speech-text bimodal language model according to the third loss.

[0027] According to a speech synthesis method provided by the present invention, the first training sample is obtained by annotation based on the following steps:

[0028] Input any speech data to be labeled into the tone labeling model, and obtain the tone description text output by the tone labeling model that describes the non-semantic information of the speech data to be labeled.

[0029] Obtain the semantic text corresponding to any of the unannotated speech data;

[0030] The tone description text is used as the first tone description text sample, the semantic text is used as the first text sample to be synthesized, and any unannotated speech data is used as the first speech tag corresponding to the first text sample to be synthesized, to construct any first training sample.

[0031] According to a speech synthesis method provided by the present invention, before using the tone description text as the first tone description text sample, the method further includes:

[0032] The tone description text is input into a large text language model to obtain a new tone description text output by the large text language model.

[0033] According to a speech synthesis method provided by the present invention, the following training steps for the speech-text bimodal language model are iteratively executed using a second training dataset until a preset termination condition is reached, thereby obtaining the tone annotation model:

[0034] Each training sample in the second training dataset includes a target speech sample, a semantic text sample corresponding to the target speech sample, and a tone description text sample that describes the non-semantic information of the target speech sample.

[0035] Input a preorder lexical subsequence from the lexical sequence corresponding to the target speech sample, the semantic text sample, and the tone description text sample in any of the training samples into the speech-text bimodal language model, and obtain the next predicted lexical from the preorder lexical subsequence output by the speech-text bimodal language model;

[0036] A fine-tuning loss is determined based on the difference between the next predicted lexical and the true lexical of the preceding lexical subsequence, in order to optimize the model parameters of the speech-text bimodal language model according to the fine-tuning loss.

[0037] According to a speech synthesis method provided by the present invention, the second training dataset is constructed based on the following steps:

[0038] Acquire multiple raw speech data and semantic text samples corresponding to each raw speech data;

[0039] Based on at least one of the classification results of fundamental frequency, energy, speech rate or emotion, select multiple target speech samples with obvious tone features from all the original speech data;

[0040] Each target speech sample is annotated with tone to generate a tone description text sample that describes the non-semantic information of the target speech sample;

[0041] Each target speech sample, the corresponding semantic text sample, and the tone description text sample are used as a training sample in the second training dataset.

[0042] According to a speech synthesis method provided by the present invention, the following training steps for a language recognition model are iteratively executed using a third training dataset until a preset termination condition is reached, thereby obtaining the speech-text bimodal language model:

[0043] Each training sample in the third training dataset is a mixed word sequence consisting of text words and / or audio words, wherein the text words and audio words are defined in the same vocabulary.

[0044] Retrieve any training sample from the third training dataset, take any preceding word subsequence from the mixed word sequence as input to the language recognition model, and obtain the next predicted word from the preceding word subsequence output by the language recognition model.

[0045] Based on the difference between the next predicted lexical and the next true lexical of any preceding lexical subsequence in the mixed lexical sequence, a pre-training loss is determined to optimize the model parameters of the language recognition model according to the pre-training loss.

[0046] According to a speech synthesis method provided by the present invention, the non-semantic information includes at least one of the following described in natural language: emotional state, speaking style, non-verbal response, anthropomorphic voice, speaking rhythm, and age information.

[0047] The present invention also provides a speech synthesis device, comprising:

[0048] The information acquisition module is used to acquire the text to be synthesized and the tone description text, wherein the tone description text is used to describe the non-semantic information of the target speech signal to be synthesized.

[0049] The text encoding module is used to jointly encode the text to be synthesized and the tone description text to obtain a mixed word sequence;

[0050] The speech generation module is used to input the mixed word sequence into the mood-controlled synthesis model and obtain the audio word sequence output by the mood-controlled synthesis model.

[0051] The speech decoding module is used to decode the audio word sequence to obtain the target speech signal.

[0052] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described speech synthesis methods.

[0053] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech synthesis method as described above.

[0054] The speech synthesis method, apparatus, electronic device, and storage medium provided by this invention, by using natural language text describing tone as an additional input parameter, enable the model to directly understand and precisely control the non-semantic attributes of speech. This solves the technical problems of existing technologies that rely on fixed labels, have coarse control granularity, and limited expressive capabilities, and significantly improves the controllability, diversity, and anthropomorphic expressiveness of synthesized speech. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating the speech synthesis method provided by the present invention.

[0057] Figure 2 This is one of the flowcharts illustrating the training process of the tone control synthesis model after reinforcement learning provided by the present invention.

[0058] Figure 3 This is the second schematic diagram of the training process after performing reinforcement learning on the tone control synthesis model provided by the present invention.

[0059] Figure 4 This is a schematic diagram of the process of obtaining a tone control synthesis model by synchronously training a speech-text bimodal language model, as provided by the present invention.

[0060] Figure 5 This is a schematic diagram of the process for training a bimodal language model of speech and text based on a first training sample, provided by the present invention.

[0061] Figure 6 This is a schematic diagram of the process for training a bimodal language model of speech and text based on a second training sample, provided by the present invention.

[0062] Figure 7 This is a schematic diagram of the process for training a bimodal language model of speech and text based on a third training sample, as provided by the present invention.

[0063] Figure 8 This is a schematic diagram of the construction process of the first training sample provided by the present invention.

[0064] Figure 9 This is a schematic diagram of the process of training a speech-text bimodal language model to obtain a tone annotation model, as provided by the present invention.

[0065] Figure 10 This is a schematic diagram illustrating the training of the tone annotation model using the second training dataset.

[0066] Figure 11 This is a schematic diagram illustrating the construction process of the second training dataset provided by the present invention.

[0067] Figure 12 This is a schematic diagram of the process of training a speech-text bimodal language model provided by the present invention.

[0068] Figure 13 This is a schematic diagram of the speech synthesis device provided by the present invention.

[0069] Figure 14 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0071] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0072] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0073] Traditional text-to-speech (TTS) methods mainly use a single parameter to control attributes such as emotion and style. Specifically, they use pre-defined speaker and recording style data for training. Each type of training data corresponds to a fixed label, and the speaker of the synthesized speech is controlled by the label ID. In the speech synthesis inference stage, only a fixed ID can be given for synthesis, and free combination is not possible. The requirements for training data are also relatively high.

[0074] With the development of speech synthesis technology, although the attribute control of speech synthesis has evolved from single parameter adjustment to multi-dimensional collaborative control based on semantic understanding, many shortcomings still exist:

[0075] 1) Existing technologies control the emotion, prosody, and intensity information of speech, but for the sake of simple reasoning, they all control the attribute labels based on the whole sentence, lacking fine-grained control capabilities, and the attribute descriptions cannot be freely combined, lacking collaborative control capabilities in multi-dimensional attribute fusion.

[0076] 2) The tone training data is prone to crosstalk. The tone training data has the problem of "binding" text, speaker and attribute. When the training data is biased, the model will infer tone label based on speaker information or text information, resulting in low control over the prompt description text.

[0077] 3) Existing attribute-controlled or tone-controlled speech synthesis methods do not take into account the understanding ability of the synthesis model, which may result in the synthesis model not understanding the attribute or tone prompt description text well, and the tone synthesis effect not matching the text description well or the effect being poor.

[0078] In view of this, the present invention provides a novel approach to speech synthesis with fine-grained tone control. By introducing non-semantic information of fine-grained tone description as a prompt during the speech synthesis process, precise control of tone attributes can be achieved. This fine-grained tone description can encompass non-semantic information such as emotional state (happy, excited, sad, frustrated, etc.), speaking style (including volume, speaking speed, whispering, shouting, etc.), non-verbal responses and anthropomorphic sounds (such as laughter, clapping, etc.), speaking rhythm (such as short pauses, long pauses, intonation, fluctuating speaking speed, etc.), and age information (such as elderly, middle-aged, youth, teenager, etc.), enabling fine-grained control of synthesized tone and effectively overcoming the control bottlenecks of traditional speech synthesis.

[0079] The following is combined with Figures 1-14 This invention describes the speech synthesis method, apparatus, electronic device, and storage medium provided by the present invention.

[0080] The speech synthesis method provided in this invention can be executed by a terminal device with a speech synthesis application installed, such as a smartphone, tablet computer, in-vehicle central control unit, smart speaker, or tablet learning machine; or it can be a cloud server or server cluster. The executing entity executes the speech synthesis method provided in this invention by running a corresponding computer program. For ease of description, the executing entity will be referred to as the system in the following embodiments.

[0081] Figure 1 This is a flowchart illustrating the speech synthesis method provided by the present invention, as shown below. Figure 1 As shown, including but not limited to the following steps:

[0082] Step 11: Obtain the text to be synthesized and the tone description text, wherein the tone description text is used to describe the non-semantic information of the target speech signal to be synthesized.

[0083] Specifically, the text to be synthesized refers to the text content that is desired to be broadcast in speech form, such as "The weather is really nice today." The tone description text is a key input in this embodiment. It is not a traditional, preset tag ID or structured parameter, but rather a text that describes the various non-semantic information that the target speech should possess in natural language form.

[0084] Non-semantic information can be understood as all phonetic features in a speech signal that, apart from the literal semantic content it conveys, reflect the speaker's state, emotion, and / or style. For example, non-semantic information may include a combination of one or more dimensions such as emotional state (e.g., happy, sad, excited), speaking style (e.g., whispering, shouting, whispering), nonverbal responses and anthropomorphic sounds (e.g., with laughter, sighing), speaking rhythm (e.g., short pauses, long pauses, fluctuating speaking speed), and age information (e.g., elderly voice, youthful voice).

[0085] In a specific implementation scenario, the system can acquire the text to be synthesized and the tone description text in various ways. For example, it can provide a human-computer interaction interface, allowing users to provide the text to be synthesized and the tone description text respectively through keyboard input or voice input. For a concrete example, a user can input the text to be synthesized as "Our team won the Best Project Award of the Year," and simultaneously input a tone description text as "Please speak in a tone of great excitement and enthusiasm, but trying to remain restrained; the overall speaking speed can be slightly faster, with a slight upward inflection at the end of the sentence." Step 11 acquires these two text pieces of information for subsequent processing. In other implementation scenarios, these two text pieces can also be automatically generated and provided by other programs through an Application Programming Interface (API).

[0086] This invention introduces natural language text describing tone as prompt input, providing a prerequisite for subsequent fine-grained and precise control of speech attributes.

[0087] Step 12: Jointly encode the text to be synthesized and the tone description text to obtain a mixed word sequence.

[0088] In this embodiment, after acquiring the text to be synthesized and the tone description text, in order for the subsequent tone control synthesis model to understand and process this textual information, it is converted into a digital format that the model can process. The core of this step is to map the independent text to be synthesized and the tone description text together into a unified representation space that can reflect their inherent relationship.

[0089] Specifically, the specific implementation method of joint coding can be:

[0090] First, using a unified tokenizer, both the text to be synthesized and the tone description text are segmented into a series of basic, discrete tokens.

[0091] Subsequently, the lexical sequences corresponding to the text to be synthesized and the tone description text are concatenated or combined in other preset ways to form a single hybrid lexical sequence. For example, the lexical sequence describing the tone of the tone description text can be placed before the lexical sequence describing the content of the text to be synthesized, forming a long sequence. In this hybrid lexical sequence, the lexical sequences describing the tone and the lexical sequences describing the content become equal modeling units, enabling the subsequent tone control synthesis model to collaboratively understand both within a unified framework.

[0092] Step 13: Input the mixed word sequence into the mood-controlled synthesis model to obtain the audio word sequence output by the mood-controlled synthesis model.

[0093] The tone control synthesis model used in this embodiment can be a pre-trained deep neural network model, which has the ability to generate corresponding speech signals based on the input text information. In a specific implementation scenario, this tone control synthesis model is specially designed and trained to deeply understand the information contained in the tone description part of the mixed word sequence and reflect it in the final generated audio word sequence and even the target speech signal.

[0094] After receiving the mixed word sequence, the mood-controlled synthesis model begins the speech synthesis process.

[0095] In a specific implementation scenario, this tone control synthesis model uses an auto-regressive approach to generate audio word sequences. An audio word sequence refers to a sequence composed of multiple discrete numerical symbols representing basic sound units. Specifically, the auto-regressive generation method can be understood as follows: the tone control synthesis model predicts the first audio word based on the complete mixed word sequence; then, this generated audio word is added to the input of the tone control synthesis model to predict the second audio word; this process is repeated, generating all audio words one after another until an end marker is generated or a preset length is reached, ultimately resulting in a complete audio word sequence.

[0096] Step 14: Decode the audio word sequence to obtain the target speech signal. Considering that the audio word sequence generated in step 13 is a digital, discrete intermediate representation, this embodiment converts it into a continuous audio waveform and outputs it.

[0097] Specifically, decoding audio word sequences can be accomplished using the neural network module of a vocoder. The vocoder receives the audio word sequence as input and synthesizes this series of discrete symbols into a continuous speech waveform that sounds natural and of high quality, i.e., the final target speech signal, which can be played directly through devices such as speakers.

[0098] The speech synthesis method provided by this invention uses natural language text describing tone as an additional input parameter, enabling the model to directly understand and precisely control the non-semantic attributes of speech. This solves the technical problems of existing technologies that rely on fixed labels, have coarse control granularity, and limited expressive capabilities, and significantly improves the controllability, diversity, and anthropomorphic expressiveness of synthesized speech.

[0099] Figure 2This is one of the flowcharts illustrating the training process of the tone control synthesis model after performing reinforcement learning, as provided by this invention. Figure 2 As shown, this includes, but is not limited to, after acquiring the target speech signal, performing reinforcement learning and post-training on the tone control synthesis model;

[0100] The post-reinforcement learning training includes iteratively performing the following steps:

[0101] Step 21: Input the target speech signal into the tone annotation model and obtain the tone-understanding text output by the tone annotation model that performs tone understanding on the target speech signal;

[0102] Step 22: Determine a reward signal based on the semantic similarity between the tone-understanding text and the tone-description text.

[0103] Step 23: Based on the reward signal, optimize the model parameters of the tone control synthesis model.

[0104] A key technical problem this invention aims to solve is to achieve fine-grained speech tone control during speech synthesis, going beyond simply controlling entire sentences. Different clauses and even different words can be controlled using prompts corresponding to tone description text, effectively realizing fine-grained speech tone control synthesis capabilities. Fine-grainedness refers to not being limited to fixed emotion tags or descriptions, but using tone description text expressed in natural language to provide a fine-grained description of the tone of the target speech signal. For example, the tone description text could be, "Synthesize an audio clip that reflects the excitement and exhilaration of winning an award, with a relatively fast overall speaking speed and a laughing tone."

[0105] Specifically, in order to further improve the tone control synthesis model's ability to understand and execute tone description text, and to solve the problems of mismatch between synthesis effect and tone description text and low control success rate in existing technologies, a scheme for post-training optimization of tone control synthesis model is provided.

[0106] The reinforcement learning post-training proposed here is a closed-loop, adaptive optimization process. It constructs a "generation-evaluation-feedback-optimization" cyclic mechanism. By calculating the semantic similarity between the tone-understanding text corresponding to the target speech signal and the input tone-description text, a quantifiable reward signal is determined to objectively evaluate the output quality of the tone-controlled synthesis model. Based on the evaluation results, the model is continuously calibrated and improved. This cyclic mechanism effectively overcomes potential biases or crosstalk problems in the training data (e.g., a certain tone in the training data always happens to be associated with a particular speaker). It controls the tone-controlled synthesis model to learn to strictly follow the input tone-description text to generate speech, rather than relying on spurious associations in the input, thereby greatly improving the accuracy and generalization ability of tone control.

[0107] Figure 3 This is the second schematic diagram of the training process after performing reinforcement learning on the tone control synthesis model provided by this invention, combined with... Figure 2 and Figure 3 As shown, the post-reinforcement learning training may include iteratively performing the following steps:

[0108] In step 21, a tone annotation model is introduced as a reward model within the reinforcement learning framework to evaluate the output of the tone-controlled synthesis model. Specifically, the tone annotation model can be a pre-trained deep neural network model with speech understanding capabilities. Its function is to receive a speech signal as input and output a tone-understood text describing the non-semantic information of the speech signal. Its function corresponds to the text-to-speech generation function of the tone-controlled synthesis model and can be regarded as a reverse mapping process from speech to text description.

[0109] After receiving the target speech signal, the tone annotation model extracts and analyzes its features and outputs tone-understanding text, which can be regarded as an objective description of the non-semantic information embodied in the target speech signal.

[0110] In step 22, the consistency between the actual output effect and the expected effect of the tone-controlled synthesis model is quantitatively evaluated. Specifically, an objective evaluation of the tone-controlled synthesis model is achieved by calculating the difference between the tone-understanding text and the tone-description text corresponding to the expected synthesis effect.

[0111] In a specific implementation scenario, the semantic similarity between the tone-understanding text and the tone-description text can be calculated, that is, at the semantic level, the meanings of the tone-understanding text and the tone-description text are compared to see if they are close.

[0112] A reward signal can be determined based on the calculated semantic similarity. This reward signal is a scalar value used to quantitatively represent the performance of the tone-controlled synthesis model in this speech synthesis. For example, the semantic similarity value can be directly used as the reward signal; a larger reward signal value indicates higher speech synthesis performance. In some embodiments, the semantic similarity value can be scaled or transformed as necessary.

[0113] In step 23, after obtaining the reward signal used for the performance evaluation of this speech synthesis, it can be used to guide the optimization and updating of the model parameters of the tone control synthesis model.

[0114] The implementation method of optimizing and updating the model parameters of the tone control synthesis model based on reward signals can be carried out using reinforcement learning algorithms, such as Proximal Policy Optimization (PPO) or Direct Preference Optimization (DPO). Their basic principle can be understood as follows:

[0115] If a speech synthesis action receives a large reward signal, the model parameters are adjusted to increase the probability of producing the same action under similar conditions in the future; conversely, if a low reward signal is received, the model parameters are adjusted to reduce the probability of the action occurring.

[0116] It should be noted that the reinforcement learning post-training used in this embodiment is a continuous iterative process, meaning that steps 21 to 23 above are repeated at least once after each generation of the target speech signal. In this way, through extensive iterative optimization, the tone control synthesis model can gradually learn to more accurately interpret and execute the synthesis instructions corresponding to various tone description texts, making its output target speech signal closer to the user's expected tone description text.

[0117] The success rate of existing synthesis models in controlling tone is not high. This is mainly because the synthesis model is prone to getting stuck in the tone with the most data distribution. When synthesizing the tone, the tone control prompt is insufficient, resulting in the synthesis model synthesizing a normal tone for most tone control prompts.

[0118] The speech synthesis method provided by this invention introduces a closed-loop post-training mechanism based on reinforcement learning and uses an independent tone annotation model as an evaluation module. This allows for dynamic and adaptive optimization of the tone control synthesis model, effectively solving the problems of insufficient control and inconsistent synthesis effect with instructions caused by data deviation in the tone control synthesis model. This significantly improves the success rate, accuracy, and generalization ability of the speech synthesis system in fine-grained tone control.

[0119] As an optional embodiment, refer to Figure 3 As shown, the step 21 mentioned above, which involves determining a reward signal based on the semantic similarity between the tone-understanding text and the tone-description text, may include, but is not limited to:

[0120] First, the tone understanding text and the tone description text are respectively input into the text encoding model to obtain the first hidden layer semantic vector and the second hidden layer semantic vector output by the text encoding model.

[0121] The text encoding model can be a deep neural network model pre-trained on a large-scale text corpus. Its core function is to map the input natural language text into a high-dimensional vector space, generating feature vectors that can capture its deep semantic information. In a preferred embodiment, the text encoding model can be a Bidirectional Encoder Representations from Transformers (BERT) model, a RoBERTa model, or other network models with similar functions in the art.

[0122] The input tone description text (denoted as T1) and the tone understanding text output by the tone annotation model (denoted as M1) (denoted as T2) are respectively fed into the text encoding model (e.g., Figure 3 The BERT model in the text encoding model processes the two texts T1 and T2, and outputs the corresponding first hidden layer semantic vector (corresponding to T1) and second hidden layer semantic vector (corresponding to T2), respectively.

[0123] The first and second hidden semantic vectors are high-dimensional, dense floating-point vectors that encode the semantic information of the text into a point in space, so that texts with similar semantics are also located close to each other in space.

[0124] Furthermore, the cosine similarity (denoted as cos) between the first hidden layer semantic vector and the second hidden layer semantic vector is used as the reward signal (denoted as reward).

[0125] After obtaining the first hidden layer semantic vector and the second hidden layer semantic vector corresponding to T1 and T2, this embodiment uses cosine similarity as a metric to calculate the degree of similarity between the first hidden layer semantic vector and the second hidden layer semantic vector.

[0126] Cosine similarity measures the directional consistency between two semantic vectors by calculating the cosine of the angle between the first and second hidden layer vectors in multidimensional space. The result of cosine similarity calculation ranges between -1 and 1. A value closer to 1 indicates that the two vectors are more aligned in direction, meaning they represent more similar text semantics; a value closer to 0 indicates that the two vectors are semantically unrelated; and a value closer to -1 indicates that the semantics are opposite. Its mathematical expression can be represented as:

[0127] ;

[0128] in, The first hidden layer semantic vector corresponding to the tone description text T1. The second hidden layer semantic vector corresponding to the text T2, which represents the tone and understanding of the text; This indicates a reward signal.

[0129] The speech synthesis method provided by this invention, by employing a text encoding model and cosine similarity, can robustly and accurately compare tone-understanding text and tone-description text at a deep semantic level, avoiding the limitations of shallow methods such as keyword matching. This ensures that the generated reward signal can truly reflect the consistency between the tone of the synthesized speech and the target instruction, thereby providing more accurate and effective optimization guidance for the reinforcement learning process, and ultimately improving the control accuracy of the tone-controlled synthesis model.

[0130] As an optional embodiment, the present invention also provides a specific method for training the above-mentioned tone control synthesis model. The aim is to use a multi-task learning framework to ensure that the final tone control synthesis model not only possesses basic speech synthesis capabilities, but more importantly, a deep understanding and precise execution capability of tone-descriptive text in natural language form. Specifically, the tone control synthesis model mentioned in the above embodiments can be trained based on the following method:

[0131] Obtain the first training dataset, and use the first training dataset to simultaneously train the speech-text bimodal language model to obtain the tone-controlled synthesis model.

[0132] The first training dataset includes at least a first training sample, a second training sample, and a third training sample. These three types of training samples each carry different training objectives and work together to affect the tone control synthesis model.

[0133] Each of the first training samples includes a first tone description text sample, a first text sample to be synthesized, and a first speech tag corresponding to the first text sample to be synthesized. The first training sample is the core data for training the tone control synthesis model to perform fine-grained tone control. Through training with this type of training sample, the tone control synthesis model can realize how to output a certain speech under what tone control prompt.

[0134] Each second training sample includes a second text sample to be synthesized and a second speech tag corresponding to the second text sample to be synthesized. The second training sample is a standard text-to-speech data pair that does not contain tone description text. It is mainly used to train the tone-controlled synthesis model to generate high-quality, natural and fluent speech signals even without a special tone control prompt.

[0135] Each of the third training samples is a plain text sample. All the third training samples contain a large amount of text information and do not involve any speech. The core function is to enhance the language understanding ability of the tone control synthesis model itself, so that it can better interpret complex and diverse tone description texts.

[0136] It should be noted that using the first, second, and third training samples to perform synchronous training of the speech-text bimodal language model does not refer to strict physical parallelism, but rather that these three types of training samples are used alternately and in combination throughout the training cycle, so that the tone control synthesis model obtained after training is synchronously improved in its ability to perform different tasks.

[0137] For example, a training batch can contain data from these three types of training samples. The speech-text bimodal language model performs different training tasks based on the different types of training samples and aggregates the gradients generated by all training tasks to gradually update the model parameters.

[0138] Through this multi-task, synchronous training framework, the final tone-controlled synthesis model will have three capabilities: 1) generating speech with specific tones by combining complex tone-descriptive text (from the first training sample); 2) generating high-quality ordinary speech (from the second training sample) when there is no tone-descriptive text; 3) having a deep understanding of tone-descriptive text (from the third training sample).

[0139] The speech synthesis method provided by this invention, by constructing a multi-task training framework containing three different types of training samples, can inject powerful natural language understanding capabilities into the tone control synthesis model during the training stage. It also takes into account fine-grained tone control capabilities and basic speech synthesis capabilities, and fundamentally solves the problem that existing synthesis models may not be able to understand complex tone control prompts, laying a solid foundation for achieving high expressiveness and high control precision in speech synthesis.

[0140] Figure 4 This is a schematic diagram of the process provided by the present invention for synchronously training a speech-text bimodal language model to obtain a mood-controlled synthesis model. Figure 4 As shown, the main steps include, but are not limited to, the following steps performed synchronously:

[0141] When the first tone description text sample, the first text sample to be synthesized, and the first preceding audio lexical sequence in the audio lexical sequence corresponding to the first speech tag are used as input to the speech-text bimodal language model, the next audio predicted lexical of the first preceding audio lexical sequence output by the speech-text bimodal language model is obtained; a first loss is calculated based on the difference between the next audio predicted lexical and the next actual audio lexical of the first preceding audio lexical sequence, so as to optimize the model parameters of the speech-text bimodal language model according to the first loss;

[0142] When the second preceding audio lexical subsequence in the audio lexical sequence corresponding to the second speech tag and the second text sample to be synthesized is used as the input of the speech-text bimodal language model, the next audio predicted lexical of the second preceding audio lexical subsequence output by the speech-text bimodal language model is obtained; a second loss is calculated based on the difference between the next audio predicted lexical and the next actual audio lexical of the second preceding audio lexical subsequence, so as to optimize the model parameters of the speech-text bimodal language model according to the second loss;

[0143] When the third preceding text word subsequence in the plain text sample is used as input to the speech-text bimodal language model, the next text predicted word of the third preceding text word subsequence output by the speech-text bimodal language model is obtained; a third loss is calculated based on the difference between the next text predicted word and the next text true word of the third preceding text word subsequence, so as to optimize the model parameters of the speech-text bimodal language model according to the third loss.

[0144] Figure 5 This is a schematic diagram illustrating the process of training a bimodal language model for speech and text based on a first training sample, as provided by the present invention. Figure 5As shown, green circles represent audio lexical units, red circles represent the start audio lexical unit in the audio lexical unit sequence, blue boxes represent a text lexical unit in the first text sample to be synthesized, orange boxes represent a descriptive text lexical unit in the first tone description text sample, and gray boxes represent text lexical units not involved in the loss calculation. In this training task with the first training sample, the speech-text bimodal language model learns how to generate target speech based on the text to be synthesized containing tone description text. The training objective is an autoregressive prediction loss, that is, to predict the next most likely audio lexical unit based on complete text information (including the first tone description text sample and the first text sample to be synthesized) and a partially generated audio lexical unit sequence (i.e., the first preceding audio lexical unit subsequence).

[0145] The first loss is calculated only when the predicted object is an audio word. That is, the loss calculation only measures the difference between the audio predicted word output by the model and the actual audio word, while the text word part does not participate in the loss calculation. This ensures that the focus of this training is on learning "how to say" rather than "what to say".

[0146] The first loss The cross-entropy loss function can be used for calculation, and its mathematical expression can be expressed as:

[0147] ;

[0148] in, N M is the length of the audio word sequence; M is the length of the text word sequence. t i Indicates the first i Each text word element; a i Indicates the first i One audio word unit; Indicates the sequence of words in a given text ( ) and before i -1 audio word unit ( Under the condition of ), generate the first i audio word The conditional probability.

[0149] The speech-text bimodal language model learns how to generate speech from text containing intonation instructions. Its training objective is an autoregressive prediction loss. The model is conditioned on complete text information (including the first intonation description text sample and the first text sample to be synthesized) and a partially generated audio word sequence (i.e., the first preceding audio word subsequence) to predict the probability distribution of the next most likely audio word.

[0150] Figure 6This is a schematic diagram of the process for training a bimodal language model of speech and text based on a second training sample, as provided by the present invention. Figure 6 As shown, the purpose of this training is to ensure the basic speech synthesis capability of the obtained tone-controlled synthesis model. Its training mechanism is similar to the training task of the first training sample, also using an autoregressive approach to predict audio word sequences. The difference is that its input condition is only the second text sample to be synthesized, without containing any tone-description text. Its training objective and second loss... The calculation method can be compared with the first loss. To maintain consistency, the loss is calculated only when predicting audio lexical units.

[0151] Figure 7 This is a schematic diagram illustrating the process of training a bimodal language model for speech and text based on a third training sample, as provided in this invention. Figure 7 As shown, the third training sample is used for a task completely unrelated to audio, with the goal of improving the natural language understanding ability of the resulting tone-controlled synthesis model. In this task, the speech-text bimodal language model takes the first half of a text (i.e., the third preorder text word subsequence) as input to predict the next most likely text word.

[0152] Third loss The cross-entropy loss is calculated by comparing the difference between the predicted words in the model's output and the actual words in the next text. It can be achieved using the standard language model cross-entropy loss, which can be expressed mathematically as follows:

[0153] .

[0154] in, Indicates giving before t -1 text units ( Under the condition of ), generate the first i Text words The conditional probability.

[0155] Considering that existing attribute-controlled or tone-controlled synthesis schemes generally fail to take into account the synthesis model's own comprehension capabilities, resulting in poor understanding of complex natural language instructions and poor synthesis effects, the speech synthesis method provided by this invention clearly distinguishes and defines three parallel training tasks. In particular, it innovatively introduces a pure text language model training task, forcing the speech-text bimodal language model to deeply learn the syntax and semantics of the language itself while learning speech synthesis. This directly enhances the tone-controlled synthesis model's ability to understand tone-descriptive text, solving the technical pain point of existing technologies that fail to understand text control prompts, leading to control failures or poor effects. As a result, the final tone-controlled synthesis model can more accurately transform complex text descriptions into nuanced speech expressions.

[0156] Existing technologies rely heavily on manually annotated, high-quality studio recording data for speech synthesis of specific styles or emotions. This data is costly to acquire, limited in scale, and typically coarse-grained in its annotation, which significantly restricts the expressiveness and generalization ability of speech synthesis models. The data annotation method proposed in this embodiment aims to produce high-quality data for fine-grained tone control training at low cost and on a large scale through an automated process.

[0157] Figure 8 This is a schematic diagram of the construction process of the first training sample provided by the present invention, as shown below. Figure 8 As shown, the first training sample was obtained by labeling based on the following steps:

[0158] Step 41: Input any speech data to be labeled into the tone labeling model, and obtain the tone description text output by the tone labeling model that describes the non-semantic information of the speech data to be labeled.

[0159] The tone annotation model mentioned here is the model that plays the role of the reward model in reinforcement learning as mentioned in any of the above embodiments. It has powerful speech understanding capabilities and can extract and annotate non-semantic information such as emotion, style and rhythm in the input speech.

[0160] The speech data to be annotated can be a vast and diverse collection of publicly available speech resources obtained from the internet, such as audiobooks, film and television clips, radio dramas, and short video dubbing. This data typically contains only speech and its corresponding text, but lacks fine-grained tone descriptions. In this embodiment, a tone annotation model is used to perform batch tone annotation processing on this raw, tone-less speech data.

[0161] After receiving a segment of speech data to be labeled, the tone annotation model analyzes its acoustic features and outputs a tone description text that describes its non-semantic information. For example, when given a segment of speech in which an actor expresses surprise, the tone annotation model might output a natural language description such as "the voice is high-pitched, the speech rate is fast, and there are obvious emotions of joy and surprise."

[0162] Step 42: Obtain the semantic text corresponding to any of the speech data to be labeled.

[0163] Semantic text is the text content broadcast by the speech data to be labeled. It can be obtained in various ways. For example, if the speech data to be labeled comes from a video with subtitles, the subtitles can be extracted directly; or, it can be obtained by transcribing the speech data to be labeled using a high-precision Automatic Speech Recognition (ASR) model.

[0164] Step 43: Use the tone description text as the first tone description text sample, the semantic text as the first text sample to be synthesized, and any unannotated speech data as the first speech tag corresponding to the first text sample to be synthesized, to construct any of the first training samples.

[0165] After obtaining the automatically generated tone description text, the speech data to be labeled, and its corresponding semantic text, these three can be combined to form a complete first training sample. The data structure of the first training sample can be set as a triple of <tone description text, semantic text, speech>.

[0166] By repeatedly executing the above steps, massive amounts of speech data without fine-grained annotations can be automatically transformed into a large-scale, high-quality first training sample set that can be used to train the intonation control synthesis model.

[0167] The speech synthesis method provided by this invention utilizes a pre-trained tone annotation model to automatically annotate massive amounts of speech data, eliminating the heavy reliance on expensive and time-consuming manual annotation data. This enables the efficient construction of large-scale datasets required for fine-grained tone control training at extremely low cost. This not only solves the industry pain point of scarce training data but also allows the final trained tone control synthesis model to learn richer and more diverse tone expressions, significantly improving its generalization ability and expressiveness.

[0168] In some cases, the tone description text directly generated by the tone annotation model may have a relatively fixed or mechanical language style. To enable the subsequent tone control synthesis model to understand and execute more diverse and colloquial natural language instructions, a text rewriting step can be added before constructing the first training sample. For example, by introducing an independent large text language model, the tone description text directly generated by the tone annotation model can be polished to improve the quality and diversity of the training data.

[0169] Specifically, before using the tone description text as the first tone description text sample, the method further includes:

[0170] The tone description text is input into a large text language model to obtain a new tone description text output by the large text language model. The large text language model used can be a large language model or other network model with text generation and rewriting capabilities; this embodiment does not specifically limit this.

[0171] In a specific implementation scenario, the tone annotation model may generate a relatively formatted tone description text for a piece of input speech data to be annotated, such as: "Emotion: happy; Speech rate: fast; Volume: high".

[0172] In this embodiment, the descriptive text is provided as input to the text-based large language model, and may be accompanied by a rewriting instruction, such as "Please rewrite the following description into one or more fluent and vivid natural language sentences." After receiving the input, the text-based large language model will utilize its language organization and generation capabilities to output one or more new descriptive texts.

[0173] For example, given the above input, the text-based large language model might output the following different styles of neo-modal descriptive text:

[0174] Version 1: "She sounded very happy, spoke very quickly, and had a loud voice."

[0175] Version 2: "This voice message is filled with joy. The speaker is in high spirits and speaks very fast, as if announcing good news."

[0176] Version 3: "A vibrant voice that conveys genuine joy from the speaker."

[0177] After acquiring the new, refined tone description text output by the large text language model, this text is used as the first tone description text sample, which, together with the corresponding semantic text and speech data, forms the first training sample. In this way, even the same speech can correspond to multiple tone description text samples with different expression styles but the same semantics in the first training dataset, greatly enriching the diversity of the training data.

[0178] The speech synthesis method provided by this invention introduces a large text language model to process and refine the automatically generated tone description text, which can significantly improve the diversity and naturalness of the training data. This enables the final trained tone control synthesis model to better generalize, understand and execute various styles of more colloquial natural language instructions, rather than being limited to a certain fixed description pattern, thereby further enhancing the robustness and user experience of the speech synthesis system.

[0179] The tone annotation model in this invention plays a crucial reward model role in the post-reinforcement learning training process, and it is also a core tool for achieving automated data annotation. Therefore, how to train a high-quality tone annotation model is of great importance to the overall technical effectiveness of this invention.

[0180] Figure 9 This is a flowchart illustrating the process of training a speech-text bimodal language model to obtain a mood annotation model, as provided by the present invention. Figure 9 As shown, the following training steps for the speech-text bimodal language model can be iteratively executed using the second training dataset until a preset termination condition is reached, thereby obtaining the tone annotation model:

[0181] Each training sample in the second training dataset includes a target speech sample, a semantic text sample corresponding to the target speech sample, and a tone description text sample that describes the non-semantic information of the target speech sample.

[0182] Input a preorder lexical subsequence from the lexical sequence corresponding to the target speech sample, the semantic text sample, and the tone description text sample in any of the training samples into the speech-text bimodal language model, and obtain the next predicted lexical from the preorder lexical subsequence output by the speech-text bimodal language model;

[0183] A fine-tuning loss is determined based on the difference between the next predicted lexical and the true lexical of the preceding lexical subsequence, in order to optimize the model parameters of the speech-text bimodal language model according to the fine-tuning loss.

[0184] The speech-text bimodal language model is an initial model to be trained. Its architecture enables it to understand information from both speech and text modalities simultaneously. In this invention, it serves not only as the base model for training the tone annotation model but also as the base model for training the tone control synthesis model.

[0185] Figure 10 This is a diagram illustrating the training of the tone annotation model using the second training dataset, combined with... Figure 9 and Figure 10 As shown, the first step is to construct a high-quality second training dataset containing manually tagged descriptive text samples.

[0186] During one forward propagation in the training process, an autoregressive prediction method is adopted. That is, after the speech-text bimodal language model to be trained receives complete speech information (i.e., target speech sample), complete text content information (i.e. semantic text sample), and incomplete tone description information (i.e., a preceding word subsequence in the word sequence corresponding to the tone description text sample), it predicts the next word most likely to follow based on this rich, cross-modal contextual information. That is, the training objective is set as the model predicting words in the tone description text.

[0187] After obtaining the next predicted word from the speech-text bimodal language model, the parameters of the speech-text bimodal language model are updated by backpropagation, using the difference between the next predicted word and the real word in the preceding word subsequence as the loss.

[0188] The difference between the next predicted word and the true word in the preceding word subsequence can be measured using the cross-entropy loss function, the mathematical expression of which is:

[0189] ;

[0190] in, The calculated cross-entropy loss; The first word in the text sample word sequence with tone description i Each word element; The first word in the target speech sample word sequence i Each word element, N It is the length of the target speech sample word sequence; It is the first word in the semantic text sample word sequence. i Each word element, M The length of the semantic text sample word sequence; Q The length of the word sequence in the text sample with tone description; Indicates that in a given audio word sequence ( ) and before i-1 text sample word sequence with tone description ( Under the condition of ), generate the first i A text sample word with tone description The conditional probability.

[0191] It is important to emphasize that in this embodiment, the cross-entropy loss is calculated only when the prediction object is a word corresponding to a tone description text sample. This means that although the speech-text bimodal language model receives all the input information, the optimization objective is solely focused on improving the accuracy of its generated tone description text. In this way, the model's attention is directed to learning how to accurately map the heard speech and the seen text content to the correct natural language description.

[0192] By iteratively executing the above training steps on the entire second training dataset until the fine-tuning loss of the speech-text bimodal language model converges or reaches the preset number of training epochs, the capabilities of the speech-text bimodal language model are successfully fine-tuned and specialized, ultimately resulting in a high-quality mood annotation model capable of accurately understanding and describing non-semantic information in speech. The fine granularity of the trained mood annotation model is mainly reflected in the refinement and comprehensiveness of the output mood description text, supporting segmented mood descriptions rather than a single mood description text for an entire segment of speech.

[0193] The speech synthesis method provided by this invention, by adopting supervised fine-tuning and carefully designing a training objective that calculates loss only when predicting tone description text, can efficiently train a general large language model into a professional, high-precision tone annotation model. This not only provides a reliable source of reward signals for the subsequent reinforcement learning process, but also provides core technical support for the large-scale automated generation of high-quality training data.

[0194] Figure 11 This is a schematic diagram of the construction process of the second training dataset provided by the present invention, as shown below. Figure 11 As shown, the main steps include, but are not limited to, the following:

[0195] Step 61: Obtain multiple raw speech data and semantic text samples corresponding to each raw speech data;

[0196] Step 62: Based on at least one of the classification results of fundamental frequency, energy, speech rate or emotion, select multiple target speech samples with obvious tone features from all the original speech data;

[0197] Step 63: Perform tone annotation on each of the target speech samples to generate tone description text samples that describe the non-semantic information of the target speech samples;

[0198] Step 64: Use each target speech sample, the corresponding semantic text sample, and the tone description text sample as a training sample in the second training dataset.

[0199] The training of the tone annotation model relies on a second training dataset, which serves primarily as a supervised fine-tuning tool for the tone annotation model to output high-quality tone description text.

[0200] Because the construction of a second training dataset in existing technologies typically requires significant investment in costly manual annotation resources, efficiently constructing such a dataset is a crucial technical challenge in this field. Furthermore, in massive amounts of raw speech data, most of the non-semantic information features are relatively unremarkable. Directly sampling and manually annotating the raw speech data would consume substantial resources on labeling these low-value data points, potentially impacting the performance of the ultimately trained tone annotation model.

[0201] To address this issue, this embodiment provides a data filtering method to optimize the construction process of the second training dataset.

[0202] In step 61, the data sources to be filtered are prepared first. The raw voice data can be a collection of voice data collected from public channels without specific filtering. At the same time, semantic text samples corresponding to the content of each raw voice data are obtained.

[0203] In step 62, data filtering is performed on all the raw speech data collected in step 61. The purpose is to automatically identify and extract speech segments with significant features in non-semantic information from the large-scale raw speech data. Specifically, one or more of the following analysis methods can be used:

[0204] (1) Analyze the fundamental frequency curve of the speech signal. The original speech data with a larger fundamental frequency standard deviation or a wider range of variation has richer intonation. Speech data that meets the conditions can be screened out by setting a threshold related to fundamental frequency analysis.

[0205] (2) Analyze the energy envelope of the original speech data, because speech with a larger energy dynamic range may contain more expressive ways of speaking.

[0206] (3) Analyze the speech rate of the original speech data, because speech with a speech rate that deviates significantly from the normal range or with drastic changes in speech rate usually also contains stronger tone features.

[0207] (4) A pre-defined emotion classification model can also be used to perform preliminary classification of all raw speech data. Then, speech classified as non-neutral emotion category is filtered out, and most speech classified as neutral tone is excluded.

[0208] By combining one or more of the above rules, a set of target speech samples with significant features in tone and emotion can be obtained.

[0209] By introducing a multi-dimensional automated screening process before manual annotation, this invention can effectively enrich speech samples with significant non-semantic information features from massive amounts of data, thereby concentrating manual annotation resources on speech samples with higher information density. This not only improves the efficiency of constructing high-quality training data and reduces costs, but also optimizes the data distribution of the dataset finally used to train the tone annotation model, which helps to train a model that is more sensitive to various tone features and has stronger recognition capabilities.

[0210] Figure 12 This is a schematic diagram of the process for training a speech-text bimodal language model provided by the present invention, as shown below. Figure 12 As shown, the speech-text bimodal language model is obtained by iteratively training a language recognition model, mainly including but not limited to the following steps:

[0211] First, a third training dataset is constructed. Each training sample in the third training dataset is a mixed sequence of text and / or audio words, defined in the same vocabulary. The vocabulary incorporates discrete text words (such as characters, words, and symbols after word segmentation) and discrete audio words (such as discrete sound units extracted from speech waveforms through techniques such as vector quantization) into the same coding space, and assigns a unique index ID to each text word and audio word.

[0212] Based on this vocabulary, each hybrid lexical sequence can be constructed in multiple ways. For example, it can be a pure text lexical sequence, a pure audio lexical sequence, or a hybrid sequence formed by concatenating text and audio lexical sequences according to temporal or semantic logic. This design enables the speech-text bimodal language model to process and understand information from different modalities within a unified framework, treating them as equal modeling units.

[0213] Secondly, the following training steps for the language recognition model are iteratively performed using the third training dataset until a preset termination condition is reached:

[0214] Retrieve any training sample from the third training dataset, take any preceding word subsequence from the mixed word sequence as input to the language recognition model, and obtain the next predicted word from the preceding word subsequence output by the language recognition model.

[0215] A language recognition model can be a network model based on a Transformer decoder architecture, which receives an incomplete sequence of preceding words as input and predicts the next most likely word in the sequence based on the known context.

[0216] In this embodiment, since the input is a mixed word sequence, the type of the next predicted word is uncertain. It may be a text word or an audio word, and the prediction needs to be made dynamically based on the modality and content of the context.

[0217] Based on the difference between the next predicted lexical and the next true lexical of any preceding lexical subsequence in the mixed lexical sequence, a pre-training loss is determined to optimize the model parameters of the language recognition model according to the pre-training loss.

[0218] After obtaining the model's prediction results, the model parameters can be updated by calculating the pre-training loss and performing backpropagation.

[0219] The pre-training loss The calculation is based on the difference between the next predicted word in the model output and the next true word in any preceding word subsequence. For example, the cross-entropy loss function can be used to measure this difference, and its mathematical expression can be represented as:

[0220] ;

[0221] Where K is the length of the mixed word sequence; The first word in the mixed word sequence i One real word element; Indicates before i -1 real word sequence ( Under the condition of ), generate the first i real word units The conditional probability.

[0222] By iteratively performing the above training steps on the entire third training dataset, the language recognition model gradually learns to make accurate sequence predictions in mixed-modal contexts, thus possessing both strong text understanding and audio understanding capabilities, and is ultimately trained into a speech-text bimodal language model.

[0223] This invention, by constructing a unified hybrid vocabulary and employing a modality-independent autoregressive prediction task, successfully trains a general speech-text bimodal language model that can equally process and understand information from both modalities. This provides a high-performance shared foundation for subsequent fine-tuning to obtain a professional tone annotation model and a powerful tone-controlled synthesis model.

[0224] The above embodiments mention tone description text used to describe the non-semantic information of the target speech signal to be synthesized. Here, non-semantic information refers to the sum of all phonetic attributes, other than the literal text content conveyed by speech, that can carry expressive features such as the speaker's emotions, state, style, and identity. Existing technologies typically only control single or limited combinations of attributes, while this invention aims to provide a fine-grained control over multi-dimensional, freely combinable non-semantic information. Therefore, the non-semantic information in this invention includes at least one of the following described in natural language: emotional state, speaking style, nonverbal responses, anthropomorphic voice, speaking rhythm, and age information.

[0225] Emotional state refers to the type of emotion conveyed in speech. For example, the tone description text can be described as "happy," "excited," "sad," "frustrated," "angry," "worried," etc. This invention not only supports these basic emotion categories but also supports subtle descriptions of the degree of emotion, such as "slightly sad" or "ecstatic."

[0226] Speaking style refers to the specific physical attributes of speech production. For example, it can be described as control of volume, such as "whispering," "whispering," "normal volume," "shouting," "roaring," etc.; it can also be described as control of timbre, such as "hoarse voice," "clear and bright voice," etc.

[0227] Nonverbal responses and anthropomorphic sounds refer to non-lexical sounds interspersed within speech that have a semantic function. For example, they can be described as "with a laugh," "with a sigh at the end of a sentence," "making a slight cough," or "with a sob."

[0228] Speech rhythm refers to the rhythm and structure of speech over time. For example, it can be described as control over speaking speed, such as "speaking too fast", "speaking slowly word by word", or "speaking at varying speeds"; it can also be described as control over pauses, such as "a short pause after 'but'" or "long pauses between sentences to create suspense"; and it can also be described as control over the overall rhythm, such as "inflection" or "straightforward delivery".

[0229] Age information refers to the age characteristics of the speaker as reflected in the voice. For example, it can be described as "the voice of an elderly person", "the voice of a middle-aged man", "the voice of a young person", "the voice of a teenager" or "the voice of a child", etc.

[0230] It is important to emphasize that the non-semantic information dimensions listed in this embodiment are not exhaustive, and they can be freely combined and used. Users can construct a highly customized tone description text by arbitrarily and complexly combining the attributes of the above multiple dimensions through a piece of natural language. For example, an effective tone description text could be "said slowly and hoarsely in the voice of a weary middle-aged man, with sighs of helplessness in his words," which integrates information from multiple dimensions such as age, speaking style, speaking rhythm, and nonverbal responses.

[0231] By explicitly supporting fine-grained natural language description and control of non-semantic information in multiple dimensions such as emotion, speaking style, rhythm, and age, this invention greatly expands the expressive boundaries of speech synthesis systems. This makes the synthesized target speech no longer a single, rigid machine voice, but capable of exhibiting rich, diverse, and highly customized voice performances like a real person, based on complex contexts and emotional needs. This allows it to better serve application scenarios with extremely high requirements for voice expressiveness, such as audio reading, film and television dubbing, game NPC dialogue, and personalized virtual assistants.

[0232] Figure 13 This is a schematic diagram of the speech synthesis device provided by the present invention, as shown below. Figure 13 As shown, it mainly includes, but is not limited to:

[0233] Information acquisition module 81 is used to acquire the text to be synthesized and the tone description text, wherein the tone description text is used to describe the non-semantic information of the target speech signal to be synthesized;

[0234] Text encoding module 82 is used to jointly encode the text to be synthesized and the tone description text to obtain a mixed word sequence;

[0235] The speech generation module 83 is used to input the mixed word sequence into the mood-controlled synthesis model and obtain the audio word sequence output by the mood-controlled synthesis model.

[0236] The speech decoding module 84 is used to decode the audio word sequence to obtain the target speech signal.

[0237] It should be noted that the speech synthesis device provided by the present invention can execute the speech synthesis method described in any of the above embodiments during specific operation, and this embodiment will not elaborate on this.

[0238] The speech synthesis device provided by this invention uses natural language text describing tone as an additional input parameter, enabling the model to directly understand and precisely control the non-semantic attributes of speech. This solves the technical problems of existing technologies that rely on fixed labels, have coarse control granularity, and limited expressive capabilities, and significantly improves the controllability, diversity, and anthropomorphic expressiveness of synthesized speech.

[0239] Figure 14 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 14 As shown, the electronic device may include a processor 1410, a communications interface 1420, a memory 1430, and a communication bus 1440, wherein the processor 1410, the communications interface 1420, and the memory 1430 communicate with each other via the communication bus 1440. The processor 1410 can call logical instructions in the memory 1430 to execute a speech synthesis method. This method includes: acquiring a text to be synthesized and a tone description text, wherein the tone description text describes non-semantic information of the target speech signal to be synthesized; jointly encoding the text to be synthesized and the tone description text to obtain a mixed word sequence; inputting the mixed word sequence into a tone-controlled synthesis model to obtain an audio word sequence output by the tone-controlled synthesis model; and decoding the audio word sequence to obtain the target speech signal.

[0240] Furthermore, the logical instructions in the aforementioned memory 1430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0241] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the speech synthesis method provided in the above embodiments, the method comprising: acquiring a text to be synthesized and a tone description text, the tone description text being used to describe non-semantic information of the target speech signal to be synthesized; jointly encoding the text to be synthesized and the tone description text to obtain a mixed word sequence; inputting the mixed word sequence into a tone-controlled synthesis model to obtain an audio word sequence output by the tone-controlled synthesis model; and decoding the audio word sequence to obtain the target speech signal.

[0242] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the speech synthesis method provided in the above embodiments. The method includes: acquiring a text to be synthesized and a tone description text, wherein the tone description text is used to describe non-semantic information of the target speech signal to be synthesized; jointly encoding the text to be synthesized and the tone description text to obtain a mixed word sequence; inputting the mixed word sequence into a tone-controlled synthesis model to obtain an audio word sequence output by the tone-controlled synthesis model; and decoding the audio word sequence to obtain the target speech signal.

[0243] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0244] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0245] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech synthesis method, characterized in that, include: Obtain the text to be synthesized and the tone description text, wherein the tone description text is used to describe the non-semantic information of the target speech signal to be synthesized; The text to be synthesized and the tone description text are jointly encoded to obtain a mixed word sequence; The mixed word sequence is input into the mood-controlled synthesis model to obtain the audio word sequence output by the mood-controlled synthesis model; The audio word sequence is decoded to obtain the target speech signal; After acquiring the target speech signal, the process also includes performing reinforcement learning and post-training on the tone control synthesis model. The post-reinforcement learning training includes iteratively performing the following steps: The target speech signal is input into a tone annotation model to obtain tone-understanding text output by the tone annotation model that performs tone understanding on the target speech signal; wherein, the tone-understanding text is used to describe the non-semantic information of the target speech signal; A reward signal is determined based on the semantic similarity between the tone-understanding text and the tone-description text. Based on the reward signal, the model parameters of the tone control synthesis model are optimized.

2. The speech synthesis method according to claim 1, characterized in that, The step of determining a reward signal based on the semantic similarity between the tone-understanding text and the tone-description text includes: The tone understanding text and the tone description text are respectively input into the text encoding model to obtain the first hidden layer semantic vector and the second hidden layer semantic vector output by the text encoding model. The cosine similarity between the first hidden layer semantic vector and the second hidden layer semantic vector is used as the reward signal.

3. The speech synthesis method according to claim 1, characterized in that, The tone control synthesis model was trained in the following way: Obtain a first training dataset, which includes at least a first training sample, a second training sample, and a third training sample; Each of the first training samples includes a first tone description text sample, a first text sample to be synthesized, and a first speech tag corresponding to the first text sample to be synthesized; each of the second training samples includes a second text sample to be synthesized and a second speech tag corresponding to the second text sample to be synthesized; each of the third training samples is a plain text sample; The speech-text bimodal language model is trained synchronously using the first training dataset to obtain the tone-controlled synthesis model.

4. The speech synthesis method according to claim 3, characterized in that, The step of simultaneously training the speech-text bimodal language model using the first training dataset to obtain the mood-controlled synthesis model includes: When the first tone description text sample, the first text sample to be synthesized, and the first preceding audio lexical sequence in the audio lexical sequence corresponding to the first speech tag are used as input to the speech-text bimodal language model, the next audio predicted lexical of the first preceding audio lexical sequence output by the speech-text bimodal language model is obtained; a first loss is calculated based on the difference between the next audio predicted lexical and the next actual audio lexical of the first preceding audio lexical sequence, so as to optimize the model parameters of the speech-text bimodal language model according to the first loss; When the second preceding audio lexical subsequence in the audio lexical sequence corresponding to the second speech tag and the second text sample to be synthesized is used as the input of the speech-text bimodal language model, the next audio predicted lexical of the second preceding audio lexical subsequence output by the speech-text bimodal language model is obtained; a second loss is calculated based on the difference between the next audio predicted lexical and the next actual audio lexical of the second preceding audio lexical subsequence, so as to optimize the model parameters of the speech-text bimodal language model according to the second loss; When the third preceding text word subsequence in the plain text sample is used as input to the speech-text bimodal language model, the next text predicted word of the third preceding text word subsequence output by the speech-text bimodal language model is obtained; a third loss is calculated based on the difference between the next text predicted word and the next text true word of the third preceding text word subsequence, so as to optimize the model parameters of the speech-text bimodal language model according to the third loss.

5. The speech synthesis method according to claim 3, characterized in that, The first training sample was obtained by labeling based on the following steps: Input any speech data to be labeled into the tone labeling model, and obtain the tone description text output by the tone labeling model that describes the non-semantic information of the speech data to be labeled. Obtain the semantic text corresponding to any of the unannotated speech data; The tone description text is used as the first tone description text sample, the semantic text is used as the first text sample to be synthesized, and any unannotated speech data is used as the first speech tag corresponding to the first text sample to be synthesized, to construct any first training sample.

6. The speech synthesis method according to claim 5, characterized in that, Before using the tone description text as the first tone description text sample, the method further includes: The tone description text is input into a large text language model to obtain a new tone description text output by the large text language model.

7. The speech synthesis method according to claim 3, characterized in that, The following training steps for the speech-text bimodal language model are iteratively executed using the second training dataset until a preset termination condition is reached, thereby obtaining the tone annotation model: Each training sample in the second training dataset includes a target speech sample, a semantic text sample corresponding to the target speech sample, and a tone description text sample that describes the non-semantic information of the target speech sample. Input a preorder lexical subsequence from the lexical sequence corresponding to the target speech sample, the semantic text sample, and the tone description text sample in any of the training samples into the speech-text bimodal language model, and obtain the next predicted lexical from the preorder lexical subsequence output by the speech-text bimodal language model; A fine-tuning loss is determined based on the difference between the next predicted lexical and the true lexical of the preceding lexical subsequence, in order to optimize the model parameters of the speech-text bimodal language model according to the fine-tuning loss.

8. The speech synthesis method according to claim 7, characterized in that, The second training dataset was constructed based on the following steps: Acquire multiple raw speech data and semantic text samples corresponding to each raw speech data; Based on at least one of the classification results of fundamental frequency, energy, speech rate or emotion, select multiple target speech samples with obvious tone features from all the original speech data; Each target speech sample is annotated with tone to generate a tone description text sample that describes the non-semantic information of the target speech sample; Each target speech sample, the corresponding semantic text sample, and the tone description text sample are used as a training sample in the second training dataset.

9. The speech synthesis method according to claim 3, characterized in that, The following training steps for a language recognition model are iteratively executed using a third training dataset until a preset termination condition is met, resulting in the speech-text bimodal language model: Each training sample in the third training dataset is a mixed word sequence consisting of text words and / or audio words, wherein the text words and audio words are defined in the same vocabulary. Retrieve any training sample from the third training dataset, take any preceding word subsequence from the mixed word sequence as input to the language recognition model, and obtain the next predicted word from the preceding word subsequence output by the language recognition model. Based on the difference between the next predicted lexical and the next true lexical of any preceding lexical subsequence in the mixed lexical sequence, a pre-training loss is determined to optimize the model parameters of the language recognition model according to the pre-training loss.

10. The speech synthesis method according to any one of claims 1-9, characterized in that, The non-semantic information includes at least one of the following described in natural language: emotional state, manner of speaking, nonverbal responses, anthropomorphic voice, speaking rhythm, and age information.

11. A speech synthesis device, characterized in that, include: The information acquisition module is used to acquire the text to be synthesized and the tone description text, wherein the tone description text is used to describe the non-semantic information of the target speech signal to be synthesized. The text encoding module is used to jointly encode the text to be synthesized and the tone description text to obtain a mixed word sequence; The speech generation module is used to input the mixed word sequence into the mood-controlled synthesis model and obtain the audio word sequence output by the mood-controlled synthesis model. The speech decoding module is used to decode the audio word sequence to obtain the target speech signal; The device is also used for: The tone control synthesis model is trained by performing reinforcement learning. The post-reinforcement learning training includes iteratively performing the following steps: The target speech signal is input into a tone annotation model to obtain tone-understanding text output by the tone annotation model that performs tone understanding on the target speech signal; wherein, the tone-understanding text is used to describe the non-semantic information of the target speech signal; A reward signal is determined based on the semantic similarity between the tone-understanding text and the tone-description text. Based on the reward signal, the model parameters of the tone control synthesis model are optimized.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the speech synthesis method as described in any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the speech synthesis method as described in any one of claims 1 to 10.

Images