Model training method, speech synthesis method, device, equipment, medium and product

Abstract

The present disclosure provides a model training method, a speech synthesis method, an apparatus, a device, a medium and a product, and belongs to the technical field of computers. The method comprises: performing base training on an initial speech synthesis model based on first training set data to obtain an intermediate speech synthesis model; the initial speech synthesis model is used to expand first emotion annotation data into a first emotion annotation sequence according to a word granularity, and perform speech synthesis of corresponding first text data based on the first emotion annotation sequence; performing fine-tuning training on the intermediate speech synthesis model based on second training set data to obtain a target speech synthesis model; the intermediate speech synthesis model is used to expand the first emotion script data into a second emotion annotation sequence of the corresponding second text data according to the word granularity, and perform speech synthesis of the corresponding second text data based on the second emotion annotation sequence. The present disclosure can accurately drive speech synthesis in word granularity to solve the problem of lack of intra-sentence emotional changes in synthesized speech.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model training method, a speech synthesis method, a model training device, a speech synthesis device, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Emotional Text-to-Speech (E-TTS) is a controllable speech synthesis technology. Its goal is to enable synthesized speech to express differences in "emotional category, intensity, and trend of change" based on input text. In related technologies, speech synthesis is usually driven by relatively coarse-grained emotions, resulting in a lack of emotional variation within sentences and a significant gap between synthesized and realistic speech. Summary of the Invention

[0003] This disclosure provides a model training method, a speech synthesis method, a model training device, a speech synthesis device, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] In a first aspect, this disclosure provides a model training method, which includes: performing base training on an initial speech synthesis model based on a first training set of data to obtain an intermediate speech synthesis model; wherein the first training set of data includes multiple first text data and first sentiment annotation data at the sentence granularity corresponding to each first text data, and the initial speech synthesis model is used to expand the first sentiment annotation data into a first sentiment annotation sequence of the corresponding first text data at the word granularity, and perform speech synthesis of the corresponding first text data based on the first sentiment annotation sequence; and performing fine-tuning training on the intermediate speech synthesis model based on a second training set of data to obtain a target speech synthesis model; wherein the second training set of data includes multiple second text data and first sentiment script data at the segment granularity corresponding to each second text data, and the intermediate speech synthesis model is used to expand the first sentiment script data into a second sentiment annotation sequence of the corresponding second text data at the word granularity, and perform speech synthesis of the corresponding second text data based on the second sentiment annotation sequence.

[0005] Secondly, this disclosure provides a speech synthesis method, which includes: inputting target text data and target sentiment annotation data corresponding to the target text data into a target speech synthesis model to obtain corresponding target synthesized speech data, wherein the target sentiment annotation data includes at least one of a first target sentiment annotation sequence at the word level, sentiment annotation data at the sentence level, and sentiment script data at the segment level; wherein the target speech synthesis model is obtained through the above-described model training method.

[0006] Thirdly, this disclosure provides a model training device, comprising: a base training module for performing base training on an initial speech synthesis model based on a first training set of data to obtain an intermediate speech synthesis model; wherein the first training set of data includes multiple first text data and first sentiment annotation data at the sentence granularity corresponding to each first text data, and the initial speech synthesis model is used to expand the first sentiment annotation data into a first sentiment annotation sequence of the corresponding first text data at the word granularity, and perform speech synthesis of the corresponding first text data based on the first sentiment annotation sequence; and a fine-tuning training module for performing fine-tuning training on the intermediate speech synthesis model based on a second training set of data to obtain a target speech synthesis model; wherein the second training set of data includes multiple second text data and first sentiment script data at the segment granularity corresponding to each second text data, and the intermediate speech synthesis model is used to expand the first sentiment script data into a second sentiment annotation sequence of the corresponding second text data at the word granularity, and perform speech synthesis of the corresponding second text data based on the second sentiment annotation sequence.

[0007] Fourthly, this disclosure provides a speech synthesis device, which includes: a synthesis module for inputting target text data and target sentiment annotation data corresponding to the target text data into a target speech synthesis model to obtain corresponding target synthesized speech data, wherein the target sentiment annotation data includes at least one of a first target sentiment annotation sequence at the word level, sentiment annotation data at the sentence level, and sentiment script data at the segment level; wherein the target speech synthesis model is obtained through the above-described model training method.

[0008] Fifthly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the model training method or speech synthesis method described above.

[0009] Sixthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described model training method or speech synthesis method.

[0010] In a seventh aspect, this disclosure provides a computer program product comprising computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the aforementioned model training method or speech synthesis method.

[0011] The model training method provided in this embodiment of the present disclosure performs base training on an initial speech synthesis model based on a first training set of data to obtain an intermediate speech synthesis model. The first training set of data includes multiple first text data and first sentiment annotation data at the sentence level corresponding to each first text data. The initial speech synthesis model is used to expand the first sentiment annotation data into a first sentiment annotation sequence corresponding to the first text data at the word level, and to perform speech synthesis on the corresponding first text data based on the first sentiment annotation sequence. The intermediate speech synthesis model is then fine-tuned based on a second training set of data to obtain a target speech synthesis model. The second training set of data includes multiple second text data and first sentiment script data at the segment level corresponding to each second text data. The intermediate speech synthesis model is used to expand the first sentiment script data into a second sentiment annotation sequence corresponding to the second text data at the word level, and to perform speech synthesis on the corresponding second text data based on the second sentiment annotation sequence.

[0012] Therefore, by expanding the global sentiment annotation data into a word-granular sentiment annotation sequence (i.e., the first sentiment annotation data) during the base training process, the obtained intermediate speech synthesis model reserves a word-granular sentiment injection interface, providing an important foundation for achieving fine and precise control and driving of speech synthesis based on word-granular sentiment data. In the fine-tuning training, sentiment script data with a smaller sentiment annotation granularity than the global sentiment annotation data is introduced and expanded according to word granularity, thereby obtaining a relatively more accurate sentiment annotation sequence (i.e., the first sentiment annotation data). Based on this sentiment annotation sequence, the speech synthesis is driven by the sentiment injection interface reserved in the base training stage. By iteratively updating the parameters, a target speech synthesis model that can accurately drive speech synthesis with word-granular precision can be obtained.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0015] Figure 1 This is a flowchart of a model training method provided in an embodiment of the present disclosure.

[0016] Figure 2 A flowchart of a speech synthesis method provided in an embodiment of this disclosure.

[0017] Figure 3 This is a block diagram of a model training device provided in an embodiment of the present disclosure.

[0018] Figure 4 This is a block diagram of a speech synthesis device provided in an embodiment of the present disclosure.

[0019] Figure 5 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0020] Figure 6 A block diagram of another electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0023] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0025] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0026] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information in this technical solution comply with relevant laws and regulations and do not violate public order and good morals. The use of user data in this technical solution follows relevant national laws and regulations. For example, appropriate measures are taken for personal information access control; restrictions are imposed on the display of personal information; the purpose of using personal information does not exceed the scope of direct or reasonable association; and explicit identification is eliminated when using personal information to avoid precisely targeting specific individuals.

[0027] Emotional Text-to-Speech (E-TTS) belongs to the field of controllable speech synthesis technology. Its goal is to enable synthesized speech to express differences in emotion category, intensity, and trends based on input text. Common controllable methods include: using discrete emotion tags or continuous emotion vectors as conditional input; using speech style or emotion representation characteristics extracted from reference audio for conditional generation; and introducing a latent space within the model that can represent speaking style or prosody, thereby achieving a certain degree of style controllability without explicit annotation.

[0028] Global style or global sentiment representation is a common implementation method in speech synthesis. For example, sentence-level sentiment labels can be provided for each training sample, or a discourse-level style or sentiment vector can be extracted from the reference audio and used as a global vector. This global vector is then used as a conditional input into the TTS model to control the overall sentiment presentation of the entire sentence. However, when it is necessary to express dynamic changes in sentiment within a sentence (e.g., the first half of the same sentence is restrained, while the second half is intensified), sentence-level or discourse-level sentiment control often struggles to accurately locate certain words or segments, leading to a phenomenon where the entire sentence changes simultaneously. This makes it difficult to achieve stable, localized, and word-granular controllable speech synthesis.

[0029] In addition, some related technologies have proposed a locally controllable emotional speech synthesis scheme based on prosodic parameters. After text-to-speech alignment, it extracts or predicts prosodic features such as pitch, energy, and duration, and uses these features as conditional inputs to influence the speech rate, pitch trend, or energy distribution of local segments, thereby achieving a certain degree of local expression change.

[0030] Furthermore, in other related technologies, emotional speech synthesis can be achieved using time-axis emotion modeling or segmented modeling. These approaches can model emotions along the speech timeline (e.g., outputting emotion intensity or segment-level emotion distribution that changes over time), and use this time-axis emotion information to drive speech generation during training or inference, thereby obtaining synthesized speech with "emotions changing over time."

[0031] While the above technical solutions can achieve emotion-controlled speech synthesis to a certain extent, they still have the following problems:

[0032] (a) Potential problems with emotional speech synthesis schemes that address sentence-level emotional conditions.

[0033] One drawback of this type of technical solution is the limited control granularity, making it difficult to form a localizable word-level (e.g., token or span) control interface. Specifically, sentence-level solutions typically apply only a single sentiment condition globally. Therefore, in scenarios requiring different words / phrases within a sentence to have different sentiment categories / intensities, it's difficult to precisely specify the control target on the input side and stably apply it to the corresponding segments. A second drawback is the potential for training distribution mismatch when migrating from sentence-level to word-level granularity. This can cause the model to ignore local conditions. In other words, if the model is exposed to a globally consistent sentiment condition distribution for a long period during training, introducing word-level local differences can easily lead to distribution drift, causing the model to tend to ignore or average out local conditions, making it difficult to stably learn word-level sentiment control.

[0034] (ii) Potential problems with locally controllable synthesis schemes based on prosodic parameters.

[0035] Prosodic parameters are typically low-level acoustic parameters that control variables, making it difficult to provide semantic word-level control inputs for emotion categories and intensity. In other words, prosodic control often uses continuous curves or labels such as pitch, energy, and duration as control targets. Its input and output are not at the same semantic level as "word or phrase emotion categories," making it difficult to directly support word-level emotion script-based control such as "specifying a word or phrase to be more angry or more joyful."

[0036] (III) Potential problems with sentiment synthesis schemes based on time-axis sentiment modeling or segmented modeling.

[0037] In this technical approach, sentiment supervision is primarily tied to the timeline, thus lacking a stable mapping between words and audio segments. Specifically, timeline sentiment modeling typically yields "frame-level or segment-level sentiment information." If a stable correspondence cannot be established between tokens or spans in the text and audio time segments, it becomes difficult to accurately project the supervision or control of small-granular text onto the corresponding audio segments. This can easily lead to problems such as unstable word-level localization, difficulty in verification, and the inability to ensure that constraints only affect the corresponding audio segments.

[0038] In view of the above, embodiments of this disclosure provide a model training method, a speech synthesis method, a model training device, a speech synthesis device, an electronic device, a computer-readable storage medium, and a computer program product.

[0039] In this embodiment, a target speech synthesis model with word-granular emotion control capability can be obtained through base training and fine-tuning training. In base training, global emotion annotation data is expanded into emotion annotation sequences in word-granular form, and an emotion injection interface in word-granular form can be reserved in the model. In fine-tuning training, emotion script data with a smaller emotion annotation granularity is introduced, expanded into emotion annotation sequences in word-granular form, and speech synthesis is performed using the emotion annotation sequences and the emotion injection interface. This results in a target speech synthesis model that can accurately drive speech synthesis with word-granular emotion data, and synthesized speech that can achieve subtle emotional changes even within sentences.

[0040] Firstly, the present disclosure provides a model training method.

[0041] Figure 1 A flowchart illustrating a model training method provided in an embodiment of this disclosure. (Refer to...) Figure 1 The model training method may include the following steps:

[0042] Step S11: Based on the first training set data, perform base training on the initial speech synthesis model to obtain the intermediate speech synthesis model.

[0043] The first training set data includes multiple first text data and first sentiment annotation data at the sentence granularity corresponding to each first text data. The initial speech synthesis model is used to expand the first sentiment annotation data into the first sentiment annotation sequence of the corresponding first text data according to the word granularity, and to perform speech synthesis of the corresponding first text data based on the first sentiment annotation sequence.

[0044] Step S12: Fine-tune the intermediate speech synthesis model based on the second training set data to obtain the target speech synthesis model.

[0045] The second training set data includes multiple second text data and first sentiment script data at the segment granularity corresponding to each second text data. The intermediate speech synthesis model is used to expand the first sentiment script data into the second sentiment annotation sequence of the corresponding second text data according to the word granularity, and to perform speech synthesis of the corresponding second text data based on the second sentiment annotation sequence.

[0046] Therefore, the training of the target speech synthesis model can be divided into two stages: base training and fine-tuning training. Base training refers to the process of training the initial speech synthesis model into an intermediate speech synthesis model using the first training set data, while fine-tuning training refers to the process of training the intermediate speech synthesis model into the target speech synthesis model using the second training set data.

[0047] In some optional implementations, the first training set data includes multiple first text data, first reference speech data corresponding to each first text data, and first sentiment annotation data at the sentence granularity corresponding to each first text data. The first text data can correspond to a single sentence; the first reference speech data is speech data with accurate and nuanced emotional expression, which can be considered as the "standard answer" or "reference answer" used to evaluate the corresponding synthesized speech data; it can be the real speech data corresponding to the first text data; and the first sentiment annotation data is sentence-level sentiment annotation data, which can be considered as global sentiment feature data, and can include information such as the sentiment category and intensity representing the entire sentence.

[0048] For example, the first training set data includes M first text data Tj (M>1, 1≤j≤M), M corresponding first reference speech data Aj, and M sentence-level first sentiment annotation data Eg(j). The first sentiment annotation data Eg(j) corresponding to the first text data Tj can be represented as... , where cg(j) is the sentiment category and sg(j) is the sentiment intensity. In other words, cg(j) is used to characterize the overall sentiment category of the first text data Tj, and sg(j) is used to characterize the overall sentiment intensity of the first text data Tj.

[0049] As mentioned earlier, the sentiment granularity of the first sentiment annotation data used in the first training set is sentence-level. If the first sentiment annotation data is directly used to drive speech synthesis, the entire sentence will only correspond to one sentiment category and intensity, failing to reflect emotional changes within the sentence. Therefore, in the base training, the sentence-level first sentiment annotation data is expanded into a word-level first sentiment annotation sequence, thereby reserving a word-level sentiment injection interface in the model. This provides an important foundation for subsequent emotion-driven speech synthesis based on word-level sentiment, enabling the production of synthesized speech that reflects emotional changes within the sentence. Word-level sentiment management or control units are smaller and more precise than sentence-level sentiment management units.

[0050] In some optional implementations, word granularity may include token granularity; wherein, tokens may correspond to characters, words, pinyin, sub-words, etc., and this disclosure does not limit this.

[0051] In some optional implementations, expanding the sentence-level first sentiment annotation data into a word-level first sentiment annotation sequence means assigning sentiment values ​​to each word element corresponding to the first text data based on the first sentiment annotation data, determining the sentiment annotation data of each word element, and sorting these sentiment annotation data according to the word element order corresponding to the first text data to form the first sentiment annotation sequence.

[0052] For example, the first sentiment annotation data Eg corresponding to the first text data T can be represented as: Where cg represents the sentiment category and sg represents the sentiment intensity; furthermore, after segmenting the first text data T, a first word sequence composed of n first word elements in order can be obtained. Furthermore, by expanding Eg according to word granularity, the corresponding first sentiment annotation sequence can be obtained. e1 is the sentiment annotation data corresponding to the first word unit t1, e2 is the sentiment annotation data corresponding to the first word unit t2, ..., en is the sentiment annotation data corresponding to the first word unit tn, and ; n≥1, 1≤i≤n. Therefore, expanding the first sentiment annotation data at the sentence level into a first sentiment annotation sequence at the word level is essentially assigning the sentiment category and sentiment intensity at the sentence level to each word element in the sentence.

[0053] When speech synthesis is performed on the corresponding first text data based on the first sentiment annotation sequence to obtain the first synthesized speech data, a loss value can be calculated based on the first synthesized speech data and the first reference speech data. The network parameters of the initial speech synthesis model are then iteratively updated based on the loss value. After satisfying preset conditions, an intermediate speech synthesis model can be obtained. These preset conditions may include at least one of the following: the loss value is less than a preset loss threshold, the number of training iterations exceeds a preset iteration threshold, and the accuracy of the current model's processing results on the validation set meets preset threshold values.

[0054] Furthermore, during base training, the first sentiment annotation sequence at the word level can be aligned with the corresponding first reference speech data. This establishes a mapping relationship between the sentiment annotation data at each word level and the corresponding speech segment, enabling more accurate and targeted speech synthesis based on the sentiment annotation data. In other words, the mapping relationship established through alignment is equivalent to creating an alignment channel, which can accurately project the sentiment annotation data at the word level onto the corresponding audio segment, achieving precise and stable localization at the word level. This allows for sentiment control and constraint only on the target speech segment and facilitates subsequent result verification.

[0055] After obtaining the intermediate speech synthesis model with pre-embedded word-level emotion interfaces, fine-tuning of the intermediate speech synthesis model is still needed to further improve the model's accurate projection of fine-grained emotions and speech synthesis.

[0056] In some optional implementations, the second training set data used to perform fine-tuning training may include multiple second text data, second reference speech data corresponding to each second text data, and first sentiment script data at the segment granularity corresponding to each second text data. The segment granularity corresponds to a word segment, which is a segment composed of multiple consecutive, adjacent words. The second text data may correspond to a sentence. The second reference speech data is speech data corresponding to the second text data that has high accuracy and subtlety in sentiment expression; it can be considered as a "standard answer" or "reference answer" for evaluating the corresponding synthesized speech data, and it may be synthesized speech data. The first sentiment script data at the segment granularity corresponding to the second text data is essentially the same as the aforementioned first sentiment annotation data at the sentence granularity and the first sentiment annotation sequence at the word granularity; both are sentiment control data used to instruct speech synthesis, but the granularity of the sentiment annotation is different. It may carry sentiment information such as sentiment category and sentiment intensity at the segment granularity, used to control the generation of speech with corresponding sentiment expression during speech synthesis.

[0057] For example, for any given text data, the corresponding sentiment script data includes sentiment annotation data for at least one word segment, where the word segment is determined based on the text data and consists of multiple consecutive words. Correspondingly, the first sentiment script data includes sentiment annotation data for at least one word segment corresponding to the second text data. In other words, a first sentiment script data includes at least one segment annotation data, where any segment annotation data is the sentiment annotation data for a corresponding word segment.

[0058] For example, the first sentiment script data corresponding to the second text data can be characterized as (k=1,…,K), where [lk, rk] corresponds to the k-th word segment, lk is the starting index of the k-th word segment in the entire word sequence corresponding to the second text data, rk is the ending index of the k-th word segment in the entire word sequence corresponding to the second text data, ck is the sentiment category of the k-th word segment, sk is the sentiment intensity of the k-th word segment, and K is the total number of word segments.

[0059] It should be noted that in a text, multiple adjacent text units (such as Chinese characters or words) have close semantic relationships and may express the same or similar emotions. Therefore, the word units corresponding to these text units can be divided into a word unit segment and represented using the same sentiment annotation data, thereby reducing the amount of data and data processing, saving resource consumption, and taking into account the accuracy of sentiment expression.

[0060] During fine-tuning training, the intermediate speech synthesis model can leverage the word-level sentiment interface reserved during base training to further expand the first sentiment script data into a second sentiment-annotated sequence for the corresponding second text data. Furthermore, since the sentiment annotation granularity of the first sentiment script data is smaller than that of the sentence-level first sentiment-annotated data, the second sentiment-annotated sequence obtained from the first sentiment script data is more accurate. After performing speech synthesis on the corresponding second text data based on the second sentiment-annotated sequence to obtain the second synthesized speech data, a loss value can be calculated based on the second synthesized speech data and the second reference speech data. Based on the loss value, some network parameters of the intermediate speech synthesis model are iteratively updated. Once preset conditions are met, the target speech synthesis model can be obtained. These preset conditions may include at least one of the following: the loss value is less than a preset loss threshold, the number of training iterations exceeds a preset iteration threshold, and the accuracy of the current model's processing results on the validation set meets preset threshold values.

[0061] It should be noted that if an alignment channel is reserved during base training, the second sentiment annotation sequence can be aligned with the second reference speech data to obtain the corresponding alignment information. Based on this, for the intermediate speech synthesis model, during the forward processing stage, each sentiment annotation data in the second sentiment annotation sequence can be accurately applied to the synthesis of the corresponding speech segment according to the alignment information, thereby accurately and subtly driving speech synthesis. During the backward processing stage, two dimensions of loss data can be used for parameter updates. One dimension represents the loss in speech synthesis ability, and the other dimension represents the loss in the ability of emotion to drive speech, reflecting the difference between the emotion expressed at the text word level and the emotion presented by the corresponding speech segment. Through this design of loss data, the accuracy and subtlety of the intermediate speech synthesis model in synthesizing speech can be effectively improved, enhancing the emotional expression ability of the synthesized speech and making it closer to real speech.

[0062] In summary, in this embodiment, by expanding the global sentiment annotation data into a word-granular sentiment annotation sequence (i.e., the first sentiment annotation data) during the base training process, the obtained intermediate speech synthesis model reserves a word-granular sentiment injection interface, providing an important foundation for achieving fine and precise control and driving of speech synthesis based on word-granular sentiment data. In the fine-tuning training, sentiment script data with a smaller sentiment annotation granularity than the global sentiment annotation data is introduced and expanded according to word granularity, thereby obtaining a relatively more accurate sentiment annotation sequence (i.e., the first sentiment annotation data). Based on this sentiment annotation sequence, the speech synthesis is driven by the sentiment injection interface reserved in the base training stage. By iteratively updating the parameters, a target speech synthesis model that can accurately drive speech synthesis with word-granular precision can be obtained.

[0063] The model training method of this disclosure embodiment will be described in detail below.

[0064] In some optional implementations, during pedestal training, the initial speech synthesis model expands the sentence-level first sentiment annotation data into a word-level first sentiment annotation sequence. Speech synthesis is then driven based on this first sentiment annotation sequence to obtain the corresponding first synthesized speech data. After obtaining multiple first synthesized speech data sets, the first synthesized speech data and the corresponding first reference speech data are input into a preset speech synthesis class loss function for calculation. The network parameters of the initial speech synthesis model are iteratively updated based on the obtained first audio loss data until a preset first convergence condition is met. At this point, pedestal training stops, and the initial speech synthesis model at this stage is used as an intermediate speech synthesis model.

[0065] In some optional implementations, the first training set data further includes first reference speech data corresponding to each first text data. Accordingly, the initial speech synthesis model is trained on the first training set data to obtain an intermediate speech synthesis model, including: inputting multiple first text data and multiple first sentiment annotation data into the initial speech synthesis model; for any first text data, the initial speech synthesis model performs the following: word segmentation on the first text data to obtain a first word sequence, the first word sequence including multiple first word elements; expanding the first sentiment annotation data corresponding to the first text data at the word granularity to obtain an initial sentiment annotation sequence corresponding to the first word sequence, the initial sentiment annotation sequence including initial sentiment annotation data corresponding to each first word element; performing distribution differentiation processing on the initial sentiment annotation data corresponding to at least some of the first word elements in the initial sentiment annotation sequence to obtain a first sentiment annotation sequence; and performing speech synthesis based on the first sentiment annotation sequence and the corresponding first word sequence to obtain first synthesized speech data corresponding to the first text data; determining first audio loss data based on at least one first synthesized speech data and the corresponding first reference speech data; and updating the network parameters of the initial speech synthesis model based on the first audio loss data.

[0066] In other words, when expanding the first sentiment annotation data at the sentence level to the word level, we can first assign sentiment values ​​to each first word element in the first word sequence based on the first sentiment annotation data. For example, we can directly use the global first sentiment annotation data as the initial sentiment annotation data for each first word element, so as to reserve the sentiment interface or sentiment channel at the word level in the model.

[0067] In some optional implementations, the first sentiment annotation data corresponding to the first text data is expanded according to word granularity to obtain the initial sentiment annotation sequence corresponding to the first word sequence, including: determining the initial sentiment annotation data of each first word element as the first sentiment annotation data; arranging the initial sentiment annotation data of multiple first word elements according to the order of the first word elements in the first word sequence to obtain the initial sentiment annotation sequence corresponding to the first word sequence.

[0068] This processing method allows for a simple and quick way to annotate the first sentiment data for each first word element, providing an important foundation for subsequent word-level sentiment annotation.

[0069] Furthermore, if the initial speech synthesis model is exposed to globally uniform sentiment-labeled data for an extended period during training (e.g., using the same sentiment-labeled data for the entire sentence), it is equivalent to using a "globally consistent" sentiment conditional distribution as a constraint. If word-level sentiment conditional distributions with local sentiment differences are subsequently introduced, distribution drift can easily occur. This can cause the initial speech synthesis model to tend to ignore local sentiment differences or average them out during learning, making it difficult to stably learn word-level sentiment control. Therefore, it is necessary to perform distribution differentiation processing on the initial sentiment-labeled sequence, ensuring local differences in sentiment-labeled data within a sentence. This allows the initial speech synthesis model to encounter sentiment-labeled data input distributions with local sentiment differences during the base training phase, thereby learning this characteristic. In subsequent inference, it can effectively utilize this learned characteristic to achieve word-level, locally differentiated speech synthesis, resulting in emotional changes in the synthesized speech of a sentence and achieving better synthesized speech.

[0070] For example, the first sentiment annotation data Eg at the sentence granularity corresponding to the first text data T can be represented as Eg = (cg, sg), where cg is the sentiment category and sg is the sentiment intensity. Further, after word segmentation of the first text data T, a first word sequence {t1,t2,……,tn} composed of n first word elements in sequence can be obtained. The sentiment is assigned to each first word element according to Eg, thereby obtaining the corresponding initial sentiment annotation sequence E1={e11,e12, …, e1n}, where e11 is the initial sentiment annotation data corresponding to the first word element t1, e12 is the initial sentiment annotation data corresponding to the first word element t2, …, e1n is the initial sentiment annotation data corresponding to the first word element tn, and any e1i takes the value (cg,sg); n≥1, 1≤i≤n. Furthermore, by performing distribution differentiation processing on the initial sentiment annotation data of at least some of the first word units, the corresponding first sentiment annotation sequence E2={e21,e22, …, e2n} can be obtained, wherein at least some of e1i and e2i are different, including different sentiment categories and / or different sentiment intensities.

[0071] In some optional implementations, the distributed differentiation processing includes at least one of the following: random perturbation, nulling, and boundary gradation. Random perturbation refers to adding sentiment perturbation to local lexical units, such as randomly selecting a lexical unit interval [p, q] and adjusting the sentiment intensity in the initial sentiment annotation data of lexical units located within that interval by upscaling, downscaling, or setting it to neutral. Nulling refers to randomly nulling the initial sentiment annotation data of local lexical units, such as by setting a nulling mask to nullify the initial sentiment annotation data of the corresponding lexical units, allowing the model to learn the knowledge of sparse local sentiment annotation data and thus be compatible with scenarios where local sentiment labels are sparse. Boundary gradation mainly refers to applying intensity gradation to lexical units near the boundaries of lexical segments, allowing the lexical segments to smoothly transition sentiment, thereby learning a more realistic and reasonable speech expression.

[0072] It should be noted that the above methods for handling distribution differences are merely illustrative examples, and the embodiments disclosed herein do not impose any limitations on them.

[0073] In some optional embodiments, alignment capability between lexical units and speech can also be established, thereby outputting alignment information based on the alignment capability. The alignment information can be used to instruct speech synthesis or to perform supervised mapping, etc. The embodiments disclosed herein do not limit this.

[0074] In some optional implementations, after expanding the first sentiment annotation data corresponding to the first text data at the word granularity to obtain the initial sentiment annotation sequence corresponding to the first word sequence, the model training method may further include: aligning the first sentiment annotation sequence with the corresponding first reference speech data to obtain alignment information, which is used to indicate the mapping relationship between the sentiment annotation data in the first sentiment annotation sequence and the speech segments in the corresponding first synthesized speech data.

[0075] By setting alignment information, the ability to align words with audio is given, which provides an important reference for subsequent annotation and verification, and helps to improve the annotation effect.

[0076] For example, for any first text data, by alignment processing, the speech segment corresponding to each first word in the first synthesized speech can be determined, or the speech segment corresponding to multiple first words in the first synthesized speech can be determined.

[0077] The alignment operation can be implemented based on explicit duration predictor or monotonic alignment mechanism, and the corresponding alignment information can be represented by alignment vector, alignment matrix, etc. This disclosure does not limit this.

[0078] For example, an initial training set data D0 can be obtained first, which belongs to a sentence-level sentiment-annotated speech dataset. Each training sample includes text T, real speech data Arera, and sentence-level sentiment-annotated data Eg=(cg,sg), where cg represents the sentiment category and sg represents the sentiment intensity. After standardizing the initial training set data, the first training set data D1 can be obtained. The purpose of performing standardization is to improve the standardization and uniformity of the training samples, thereby improving the training effect. The standardization process includes uniform sampling rate, loudness standardization, audio denoising, audio truncation, and silence processing. One or more of the standardization processes can be selected for execution based on the available hardware and software resources and the amount of resources required to perform the standardization process. This embodiment of the disclosure does not limit this.

[0079] Secondly, the first text data in the first training set data D1 is segmented to obtain a corresponding first word sequence including multiple first word elements. The granularity of word segmentation can be characters, pinyin, or sub-words, etc., and this embodiment does not limit this.

[0080] Furthermore, the sentence-level sentiment annotation data is broadcast as a word-level sentiment annotation sequence, thereby pre-embedding a word-level sentiment interface. For example, the sentence-level sentiment annotation data Eg is mapped using a pre-defined mapping function. Converting the first sentiment annotation sequence to word granularity is equivalent to broadcasting the first sentiment annotation data to each first word element. Any sentiment annotation data ei in the resulting first sentiment annotation sequence corresponds to the i-th word element, where i=1,...,N, and N is the total number of first word elements in the current first word sequence. The mapping function... It can be implemented by means of emotion category embedding combined with intensity scalar splicing, multi-layer perceptron (MLP) projection, etc., and the embodiments disclosed herein are not limited thereto.

[0081] For any given first text data, the initial speech synthesis model can also fuse and represent the first word unit in the corresponding first word sequence with its sentiment annotation data, as shown below: Hi represents the text encoding of the first word element (e.g., token), and ei represents the sentiment annotation data of the first word element (which may have undergone distribution differentiation processing). The Fuse function can be implemented using methods such as additive fusion, gated fusion, feature-wise linear modulation (FiLM), and attention fusion; this embodiment does not limit the implementation of these methods. Based on this, the acoustic processing network structure in the initial speech synthesis model can be determined according to... The corresponding target speech is generated, and these target speech are arranged in order to obtain the first synthesized speech data corresponding to the first text data.

[0082] It should be noted that the alignment capability between words and audio can be preserved during the above training process, the alignment channel can be reserved, and the corresponding alignment information can be output as needed.

[0083] In summary, in this embodiment, through base training, even with only sentence-level sentiment annotation, the initial speech synthesis model can possess and adapt to word-level sentiment input interfaces from the early stages of training. Furthermore, through training strategies such as sentiment distribution differentiation, it can adapt to future input distributions of word-level sentiment annotation data with localized sentiment differences. This allows for a smooth transition from sentence-level sentiment control to word-level sentiment control. In other words, base training, through processes such as broadcasting global sentiment annotation data within a sentence, lexical sentiment annotation fusion, word-level sentiment input interfaces, and increasing the differentiation of sentiment annotation data distribution, provides a crucial foundation for achieving word-level sentiment-driven speech synthesis.

[0084] It should be noted that the first training set data used in the base training can be highly generalized data, with the aim of enabling the model to learn more general and extensive knowledge. However, in actual use, the model is usually applied to specific business or application scenarios. Therefore, the intermediate speech synthesis model can be further fine-tuned to obtain a better target speech synthesis model.

[0085] In some optional embodiments, a second training set of data needs to be acquired before performing fine-tuning training. In some optional implementations, this second training set of data can be constructed through word-granular sentiment script generation and synthetic data bootstrapping.

[0086] The word-granular sentiment script generation refers to generating word-granular sentiment script data for each piece of text data. A sentiment script data point may correspond to a single lexical segment within the text. This lexical segment is typically a multi-lexical sequence with closely related semantics and expressing similar or even identical sentiments. Therefore, a sentiment script data point represents the sentiment annotation data of this lexical segment. For example, for a given piece of text data, corresponding sentiment script data is generated, which may include one or more segment annotation data points. Each segment annotation data point corresponds to a lexical segment, used to represent sentiment-related data such as the sentiment category and intensity of that lexical segment.

[0087] Synthetic data bootstrapping is an iterative model enhancement and data augmentation method that uses high-quality synthetic data generated by the model itself (such as an intermediate speech synthesis model) to retrain or fine-tune the model, forming a closed loop of "generation, screening, training, and regeneration" to improve the model's capabilities at low cost and on a large scale.

[0088] In some optional implementations, before fine-tuning the intermediate speech synthesis model based on the second training set data to obtain the target speech synthesis model, the model training method may further include: acquiring a text dataset, which includes multiple third text data; generating second sentiment script data at the segment granularity corresponding to each third text data; inputting the multiple third text data and the multiple second sentiment script data into the intermediate speech synthesis model for speech synthesis to obtain third synthesized speech data corresponding to the third text data; and obtaining the second training set data based on the multiple third text data, the multiple second sentiment script data, and the multiple third synthesized speech data.

[0089] The text dataset is a large-scale text collection, which can be text data from a business system, dedicated corpus data, etc., and this disclosure does not limit it.

[0090] Therefore, it can be seen that the intermediate speech synthesis model can be fine-tuned using text data from specific scenarios or business systems. During the fine-tuning process, a second sentiment script is first generated for each third text data set. Then, the intermediate speech synthesis model performs speech synthesis based on the third text data and the second sentiment script, obtaining the third synthesized speech data. A second training set is then constructed based on this. When performing speech synthesis with reference to the second sentiment script data, since a word-level sentiment interface has already been reserved in the base training, the second sentiment script data can also be expanded into a word-level sentiment tag sequence to learn word-level sentiment-driven speech synthesis capabilities.

[0091] In some optional implementations, the sentiment script data at the fragment granularity can be generated using methods such as random sampling, template rules, or text semantic-based strategies, and this disclosure does not impose any limitations on this. For example, sentiment templates and corresponding matching rules can be pre-set, sentiment templates can be filled with text data, and the filled sentiment templates can be matched with preset sentiment script data according to the matching rules to determine the corresponding sentiment script data. For example, the text can be analyzed at the semantic level, and the corresponding sentiment script data can be determined based on the analysis results.

[0092] In some optional implementations, generating second sentiment script data at the fragment granularity corresponding to each third text data includes: performing word segmentation on any third text data to obtain a third word sequence; performing fragment sampling on the third word sequence to obtain at least one second word element fragment; performing sentiment annotation on each second word element fragment to obtain the second sentiment script data corresponding to the third text data; wherein, the second sentiment script data includes second fragment annotation data corresponding to each second word element fragment.

[0093] For example, for a certain third-party text data, after word segmentation, multiple third-party words are obtained. Based on this, K second-party word segments are obtained through fragment sampling, each of which includes multiple consecutive third-party words. An overall sentiment assignment is performed on each second-party word segment to determine its sentiment category ck and sentiment intensity sk. The sentiment intensity of the second-party word segment can be consistent with the global sentiment or locally strengthened or weakened. Alternatively, samples with local sentiment category changes can be generated according to a preset ratio. Furthermore, the sentiment intensity at the segment boundaries can be gradually adjusted to achieve a smooth transition of sentiment and improve the realism and reasonableness of the samples.

[0094] For example, the third text data corresponds to a third word sequence containing 16 third word elements. The 2nd to 5th third word elements form second word element segment 1, the 8th and 9th third word elements form second word element segment 2, and the 14th to 16th third word elements form second word element segment 3. For second word element segment 1, its second segment annotation data is determined as (c1, s1) through sentiment assignment. Similarly, for second word element segment 2, its second segment annotation data is determined as (c2, s2) through sentiment assignment. And for second word element segment 3, its second segment annotation data is determined as (c3, s3) through sentiment assignment. Based on this, the second sentiment script data corresponding to this third text is obtained.

[0095] In some optional implementations, multiple third-text data and multiple second-emotion script data are input into an intermediate speech synthesis model for speech synthesis to obtain third-synthesized speech data corresponding to the third-text data. This includes: inputting multiple third-text data and multiple second-emotion script data into the intermediate speech synthesis model; for any third-text data, the intermediate speech synthesis model performs the following: segmenting the third-text data to obtain a third-word sequence, which includes multiple third-word units; expanding the second-emotion script data corresponding to the third-text data at the word granularity to obtain a third-emotion annotation sequence corresponding to the third-text data, which includes emotion annotation data corresponding to each third-word unit; and performing speech synthesis based on the third-emotion annotation sequence and the corresponding third-word sequence to obtain the third-synthesized speech data corresponding to the third-text data.

[0096] Therefore, in this process, word granularity can be used as the emotion control unit to drive the synthesis of corresponding speech segments, so that emotional changes can be presented within a sentence, thereby making the third synthesized speech data closer to real and reasonable speech data.

[0097] In some alternative implementations, sentiment data can be expanded in word-granular form based on the second sentiment script data to obtain the corresponding third sentiment annotation sequence.

[0098] In some optional implementations, the second sentiment script data includes at least one second segment annotation data, which is used to characterize the sentiment annotation data of the corresponding second word segment. The second word segment includes multiple consecutive third words. Accordingly, the second sentiment script data corresponding to the third text data is expanded at the word granularity to obtain the third sentiment annotation sequence corresponding to the third word sequence, including: for any third word, if the third word is located in any second word segment, the second segment annotation data corresponding to the second word segment is used as the sentiment annotation data of the third word in the third sentiment annotation sequence; if the third word is not located in any second word segment, the sentence-level sentiment annotation data or the preset sentiment annotation data of the third text data is used as the sentiment annotation data of the third word in the third sentiment annotation sequence.

[0099] In other words, for any third word element in any third text data, if the third word element is located within a second word element segment, then the segment annotation data of that second word element segment can be directly used as the sentiment annotation data of the third word element in the third sentiment annotation sequence. Conversely, if the third word element is not located within any second word element segment, then the global sentiment annotation data of the third text data (such as sentence-level sentiment annotation data) can be used as the sentiment annotation data of the third word element in the third sentiment annotation sequence. Alternatively, preset sentiment annotation data can be specified in advance and used as the sentiment annotation data of the third word element in the third sentiment annotation sequence. Through this processing method, the sentiment annotation data of the third word element in the third sentiment annotation sequence can be determined conveniently and accurately.

[0100] Taking the aforementioned third text data corresponding to a third word sequence containing 16 third word elements as an example, assuming its sentence-level sentiment annotation data is (c0,s0), then based on the three second segment annotation data, it can be expanded into a word-level third sentiment annotation sequence: {(c0,s0), (c1,s1), (c1,s1), (c1,s1), (c1,s1), (c0,s0), (c0,s0), (c2,s2), (c2,s2), (c0,s0), (c0,s0), (c0,s0), (c0,s0), (c3,s3), (c3,s3), (c3,s3)}. Furthermore, for any second word element segment's segment boundary, some intensity gradients can be applied to make the sentiment changes smoother and more natural.

[0101] Based on this, in order to obtain higher quality second training set data, multiple third text data, multiple second emotion script data, and multiple third synthesized speech data can be further filtered and / or corrected.

[0102] In some optional implementations, before inputting multiple third text data and multiple second emotion script data into an intermediate speech synthesis model for speech synthesis to obtain third synthesized speech data corresponding to the third text data, the model training method may further include: deleting part of the third text data and the corresponding second emotion script data, third synthesized speech data, and / or correcting part of the second emotion script data of the third text data according to a preset quality index. The preset quality index may include confidence level.

[0103] Therefore, we can delete low-quality third-party text data and manually correct questionable second-party sentiment script data to obtain high-quality, highly certain training data.

[0104] In some optional implementations, based on preset quality indicators, some third text data and corresponding second sentiment script data and third synthesized speech data are deleted, and / or, the second sentiment script data of some third text data is corrected, including: generating prediction script data and the confidence level of prediction script data for each third text data based on a preset sentiment annotation model; deleting third text data and corresponding second sentiment script data and third synthesized speech data with confidence levels lower than a preset confidence threshold; and / or, for any third text data, performing a consistency check on the second sentiment script data and prediction script data corresponding to the third text data to determine a consistency indicator; selecting at least one third text data from multiple third text data based on the consistency indicator, and correcting the second sentiment script data corresponding to at least one third text data to obtain the corrected second sentiment script data corresponding to at least one third text data.

[0105] The preset sentiment annotation model can include a multimodal large model, a sentiment recognition model, or a combined model, and this disclosure does not limit this. The multimodal large model, sentiment recognition model, or combined model can be obtained from open-source model libraries. For example, the Whisper model from OpenAI can be obtained. Since the intermediate layer features output by the encoder of the Whisper model contain rich sentiment information, high-precision sentiment classification can be achieved through simple fine-tuning, thus obtaining a model capable of converting speech into text and adding sentiment annotation data. Alternatively, the Wav2vec2 model can be obtained, which learns general speech representations from large-scale unlabeled audio data through self-supervised learning, reducing the dependence of tasks such as speech recognition on large-scale labeled data. Based on this, a model capable of converting speech into text and adding sentiment annotation data can be obtained through simple fine-tuning. Another example is the emotion2vec model, a self-supervised pre-trained model specifically for sentiment representation. During the pre-training stage, the feature space is directly constrained by sentiment loss, thus requiring no complex fine-tuning and allowing for one-click output of text and its sentiment annotation data. Furthermore, the aforementioned deletion and correction operations can improve the quality of emotional script data and speech synthesis data, thereby enhancing the model training effect.

[0106] For example, the second sentiment script data corresponding to the third text data can be characterized as Here, lk and rk represent the start and end positions of the corresponding word segment, respectively, and ck and sk represent the sentiment category and sentiment intensity corresponding to the word segment. Furthermore, the third-party synthesized speech data and the corresponding third-party text data can be input into a general multimodal model to obtain the corresponding sentiment annotation results: PK represents the confidence level. Based on this, third text data with a confidence level less than or equal to a preset confidence threshold, along with their second sentiment script data and third synthesized speech data, can be removed from the entire dataset, or marked into an artificial pool for further manual review and correction.

[0107] For example, consistency checks can be performed on the second sentiment script data and the prediction script data from the perspectives of segment coverage, category matching degree, and intensity deviation threshold to obtain a consistency index of quantified values. Based on the values ​​of the consistency index, abnormal samples are screened out for manual review. The abnormal samples are then processed by manual review to perform boundary correction, sentiment category correction, and sentiment intensity correction, thereby forming the final accurate sentiment script data.

[0108] In summary, it is possible to generate word-level sentiment script data from large-scale text sets, synthesize speech with local sentiment variations using sentence-level base models (i.e., intermediate speech synthesis models), and then process the data through filtering and manual correction to form a bootstrap dataset. (i.e., the second training set data), where T represents the second text data, hat{A} ​​represents the second reference speech data, Eg is optional and is the sentiment annotation data at the sentence level, and the second sentiment script data S is explicitly retained as the "known sentiment script".

[0109] In the process of selecting and correcting sample data, a semi-automatic word-granularity annotation mechanism can be introduced. This involves a noise reduction process that utilizes models to automatically annotate sentiment and output confidence scores, performs consistency checks on the scripts, and uses manual verification. This process can explicitly solve problems such as weak supervision noise and boundary uncertainty at the word-granularity level, and is a key processing method for forming a second training set data capable of model training.

[0110] In some alternative implementations, the intermediate speech synthesis model can be fine-tuned after obtaining the second training set data.

[0111] In some optional implementations, the second training set data further includes second reference speech data corresponding to each second text data, and the first sentiment script data includes at least one first segment annotation data; accordingly, the intermediate speech synthesis model is fine-tuned and trained based on the second training set data to obtain the target speech synthesis model, including: inputting multiple second text data and multiple first sentiment script data into the intermediate speech synthesis model; for any second text data, the intermediate speech synthesis model performs: word segmentation of the second text data to obtain a second word sequence, the second word sequence including multiple second word elements; expanding the first sentiment script data corresponding to the second text data at the word granularity to obtain a second sentiment annotation sequence corresponding to the second word sequence, the second sentiment annotation... The annotation sequence includes sentiment annotation data corresponding to each second word; and speech synthesis is performed based on the second sentiment annotation sequence and the corresponding second word sequence to obtain second synthesized speech data corresponding to the second text data; the speech segments in the second synthesized speech data corresponding to each first segment annotation data are determined; sentiment prediction is performed on each speech segment based on a preset sentiment processing model to obtain segment sentiment prediction data corresponding to each speech segment; second audio loss data is determined based on at least one second synthesized speech data and the corresponding second reference speech data; sentiment loss data is determined based on at least one first segment annotation data and the corresponding segment sentiment prediction data; and some network parameters of the intermediate speech synthesis model are updated based on the second audio loss data and the sentiment loss data.

[0112] Therefore, it can be seen that the data processing process performed by the intermediate speech synthesis model in fine-tuning training is similar to the data processing process when obtaining the bootstrap dataset. Furthermore, since the second training set data used in fine-tuning training is more accurate and reasonable, more accurate output results can be obtained, and the model loss can be determined based on these output results to achieve model fine-tuning.

[0113] In some optional embodiments, the first segment annotation data is used to characterize the sentiment annotation data of the corresponding first word segment, the first word segment including multiple consecutive second words; accordingly, the first sentiment script data corresponding to the second text data is expanded at the word granularity to obtain the second sentiment annotation sequence corresponding to the second word sequence, including: for any second word, if the second word is located in any first word segment, the first segment annotation data corresponding to the first word segment is used as the sentiment annotation data of the second word in the second sentiment annotation sequence; if the second word is not located in any first word segment, the sentence-level sentiment annotation data or the preset sentiment annotation data of the second text data is used as the sentiment annotation data of the second word in the second sentiment annotation sequence.

[0114] Therefore, for any second word element, if it is located in a certain first word element segment, the sentiment annotation data of that first word element segment can be used as its sentiment annotation data in the second sentiment annotation sequence. If it is not located in any first word element segment, the sentiment annotation data at the sentence level or the preset sentiment annotation data can be used as its own sentiment annotation data. In this way, the second sentiment annotation sequence can be obtained conveniently and accurately.

[0115] It should be noted that after obtaining the second sentiment annotation sequence based on the above method, boundary gradation can be applied to the segment boundaries to make the sentiment changes of the first word segment near its boundary have a gradual effect, thus more closely resembling the real and reasonable sentiment expression.

[0116] In some optional embodiments, the preset emotion processing model can be an emotion recognition model, a multimodal model, or a combined model, and this disclosure does not limit this. The preset emotion processing model can predict the emotion of each speech segment in the synthesized second speech data, obtaining segment emotion prediction data. This data is then compared with the corresponding segment annotation data to determine whether the emotional expression of the synthesized speech is accurate. This data is then combined with the second audio loss data to jointly determine the loss value. Based on the loss value, some network parameters are adjusted until a second convergence condition is met, at which point fine-tuning training stops, and the intermediate speech synthesis model at this point is used as the final target speech synthesis model.

[0117] Therefore, in the fine-tuning training of the intermediate speech synthesis model, not only the second audio loss data (equivalent to the first audio loss data in the base training) used to characterize the audio synthesis capability, but also the emotional loss data used to characterize the driving ability of emotion on speech. The emotional loss data can reflect whether the model can accurately synthesize speech data that matches the emotional annotation data corresponding to the word granularity, so that the speech corresponding to a sentence can also present subtle emotional changes, thus possessing accurate and delicate speech synthesis capabilities.

[0118] In some alternative implementations, explicit supervision can be introduced when determining the loss data to enable more flexible and targeted training.

[0119] In some optional implementations, before determining the second audio loss data based on at least one second synthesized speech data and the corresponding second reference speech data, the model training method further includes: determining whether the second word element participates in the calculation of the second audio loss data and / or sentiment loss data based on the supervision identifier of the second word element; wherein, the supervision identifier is added to the second word sequence based on a preset supervision mask, and the supervision mask is of the same length as the second word sequence; if the supervision identifier is a valid value, it is determined that the second word element corresponding to the supervision identifier participates in the calculation of the second audio loss data and / or sentiment loss data; if the supervision identifier is an invalid value, it is determined that the second word element corresponding to the supervision identifier does not participate in the calculation of the second audio loss data and / or sentiment loss data.

[0120] This processing method allows for adjustment of the impact of different lexical units or different lexical segments on sentiment loss data, thereby enabling more flexible and targeted model training based on specific needs.

[0121] For example, if a second word sequence corresponding to a certain second text data includes N second word elements, a supervision mask of N bits can be pre-set. Each mask corresponds to one second word element and is used to represent the supervision identifier of the corresponding second word element. The mask specifies that a valid value for the supervision identifier is "1" and an invalid value is "0". For any second word element, if its corresponding supervision identifier is 1, then the second word element participates in the calculation of the second audio loss data and / or sentiment loss data; if its value is 0, then it does not participate in the calculation of the second audio loss data and / or sentiment loss data.

[0122] For example, for any second text data input into the intermediate speech synthesis model, it first undergoes word segmentation to obtain the corresponding second word sequence. Then, through word-granular sentiment expansion, the corresponding first sentiment script data S is expanded into a word-granular second sentiment annotation sequence E={ei}, and a corresponding supervision mask mi is generated. The value of mi indicates which second word units have explicit supervision and which second word units do not have explicit supervision, so as to calculate the second audio loss data and / or sentiment loss data in the subsequent process. Further, the text encoding module in the intermediate speech synthesis model encodes each second word unit to obtain the corresponding hi, and then fuses hi with the corresponding sentiment annotation data ei to obtain fused representation data. Based on this, loss calculation is performed on the audio synthesis capability dimension to obtain L{tts}, which is equivalent to TTS reconstruction loss. Further, if an alignment channel is reserved during the base training phase, this alignment channel can be used to establish a mapping relationship between lexical units and audio, thereby mapping each first lexical unit segment [lk,rk] to the corresponding speech segment [uk,vk]. Then, a pre-defined sentiment processing model is used to evaluate the sentiment of [uk,vk], and word-granular sentiment consistency loss is calculated based on the evaluation results to obtain the sentiment loss data L{emo}(k). Additionally, boundary smoothing loss data L{smooth} can be introduced as part of the loss data to suppress boundary sentiment abrupt changes.

[0123] In some optional embodiments, determining sentiment loss data based on at least one first segment annotation data and corresponding segment sentiment prediction data includes: for any segment sentiment prediction data, calculating an initial sentiment loss value based on the segment sentiment prediction data and the corresponding first segment annotation data; determining a loss weight corresponding to each segment sentiment prediction data based on the confidence levels of multiple segment prediction data; and performing a weighted calculation based on the loss weight and the initial sentiment loss value to obtain the sentiment loss data.

[0124] This shows that, regarding data on emotional loss Confidence levels can be used for weighted calculations, such as... k is the identifier of the first word segment or speech segment, and the loss weight wk is determined based on the confidence of the sentiment prediction data of the corresponding speech segment. Based on this, the loss value of high-confidence speech segments accounts for a relatively high proportion, while the loss value of low-confidence speech segments accounts for a relatively low proportion, which helps to improve the rationality of sentiment loss data and thus improve the model fine-tuning effect.

[0125] In summary, the loss data used during base training can be characterized as follows: The loss data used during fine-tuning training can be characterized as follows:

[0126] .in, and This is a loss adjustment factor used to adjust the proportion of different types of loss in the overall loss. By using this method of loss data calculation, the network parameters of the model can be adjusted in a targeted manner for different training stages to achieve the desired training effect.

[0127] In summary, the model training method of this disclosure includes at least the following features and their corresponding technical effects.

[0128] First, by controlling emotion at the word level, we can obtain more stable, accurate, and fine-grained speech synthesis capabilities, which can significantly reduce the risks of local emotional changes being ignored and training not converging.

[0129] The corresponding technical means include: pre-embedding word-level sentiment data input channels in the base training stage of sentence-level sentiment annotation, and unifying sentence-level sentiment annotation data into sentiment annotation sequences of the same length as word-level segmented word sequences through broadcasting and other methods; at the same time, introducing distribution differentiation processing such as local perturbation, local emptying, and boundary gradation, so that the model is exposed to the input distribution of local sentiment differences in sentences in the early stage of training.

[0130] The corresponding technical effects include: compared to "directly inserting word-level sentiment modulation into a model that has only seen global sentiment annotations", the current processing method can significantly alleviate the failure of local sentiment control and the instability of fine-tuning caused by distribution drift, making word-level sentiment control capabilities easier to form and more reproducible.

[0131] Second, it is more robust to word-level annotation noise and boundary uncertainty, which can reduce the negative impact of incorrect supervision on the model.

[0132] The corresponding technical means include: adopting a data closed loop of synthesis bootstrapping and semi-automatic annotation, that is, the synthesized speech is first annotated with sentiment by multimodal models, and the sentiment script data (corresponding to the sentiment prediction data of the segment) and confidence scores are output at the word granular level. Then, the consistency between the confidence score threshold and the known sentiment script data (i.e., the second sentiment script data) is checked to remove abnormal samples or correct abnormal samples. It also supports the processing of abnormal samples entering the artificial pool for review. Furthermore, robust strategies such as confidence weighting, boundary perturbation, and local sentiment annotation data emptying are introduced during training.

[0133] The corresponding technical means include: compared with the method of "automatically aligning the obtained word-level sentiment features and directly using them for training" in related technologies, the current processing method can significantly reduce training divergence, local mutation and control instability caused by noise labels, boundary offsets, etc., and improve the reliability and consistency of word granularity controllability.

[0134] Third, word-level supervision can be precisely mapped to corresponding speech segments, making local sentiment constraints more verifiable and localizable.

[0135] The corresponding technical means include: retaining the word speech alignment channel in the model structure during base training, and using the alignment channel to obtain alignment information during the fine-tuning stage. The alignment information is then used to map small-granular text data such as words and segments to speech segments, and then word-granular sentiment consistency loss is applied to the segment and jointly optimized with the basic speech reconstruction loss.

[0136] The corresponding technical means include: compared with the related technologies that use "text-side modulation" or "use alignment to generate word-level features, but lack fragment-level constraint loops", the current processing method can truly apply the emotion that a certain word (or fragment) should present to the constraint of the corresponding speech fragment, thereby improving the localizability, verifiability and control accuracy of local emotion control.

[0137] Fourth, the word-level sentiment control interface is more explicit and editable, supports scripting the specification of sentiment categories and intensity for segments, and supports smooth sentiment transitions at segment boundaries.

[0138] The corresponding technical means include: defining sentiment script data at the word granularity and specifying the processing mechanism for expanding it into sentiment annotation sequences at the word granularity level; introducing boundary gradation and smooth transition processing mechanisms in the generation and training of sentiment script data, and adding smooth regularization loss to the loss to suppress boundary abrupt changes.

[0139] The corresponding technical means include: compared with the implicit control method of "obtaining word-level representation from frame-level sentiment feature alignment and then modulating it through FiLM" in related technologies, the embodiments of this disclosure provide a more intuitive input form of "editable sentiment script data", which makes word and fragment-level sentiment adjustment more operable and consistent, and at the same time makes the boundary transition more natural and realistic.

[0140] Fifth, given only sentence-level sentiment annotation data, it is possible to construct large-scale training data with word-level sentiment annotation data.

[0141] The corresponding technical means include: constructing a high-quality second training set data through a closed loop of "word-granular sentiment script generation, base model synthesis bootstrapping, semi-automatic annotation screening and verification"; the first sentiment script data, as a known script, can be used for consistency verification, thereby reducing the uncontrollable noise of automatic annotation.

[0142] The corresponding technical means include: compared with the processing methods in related technologies that rely on external sentiment representation and alignment processes or require additional high-quality word-level labeled training data, the embodiments of this disclosure have a lower dependence on manual and refined sentiment labeling in terms of data sources, and are more likely to form a sustainable supply and iteration of training data in engineering.

[0143] Secondly, embodiments of this disclosure provide a speech synthesis method.

[0144] Figure 2 A flowchart illustrating a speech synthesis method provided in an embodiment of this disclosure. (Refer to...) Figure 2 The speech synthesis method may include the following steps:

[0145] Step S21: Input the target text data and the target sentiment annotation data corresponding to the target text data into the target speech synthesis model to obtain the corresponding target synthesized speech data. The target sentiment annotation data includes at least one of the following: word-level first target sentiment annotation sequence, sentence-level sentiment annotation data, and segment-level sentiment script data.

[0146] The target speech synthesis model is obtained through the model training method described in any one of the embodiments of this disclosure.

[0147] In some optional implementations, if the target sentiment annotation data includes a first target sentiment annotation sequence, speech synthesis is performed based on the first target sentiment annotation sequence; if the target sentiment annotation data includes sentence-level sentiment annotation data, the sentence-level sentiment annotation data is expanded into a second target sentiment annotation sequence at the word level corresponding to the target word sequence, and speech synthesis is performed based on the second target sentiment annotation sequence, where the target word sequence is obtained by segmenting the target text data; if the target sentiment annotation data includes sentence-level sentiment annotation data and segment-level sentiment script data, expansion processing is performed based on the sentence-level sentiment annotation data and the segment-level sentiment script data to obtain a third target sentiment annotation sequence at the word level corresponding to the target word sequence, and speech synthesis is performed based on the third target sentiment annotation sequence.

[0148] It should be noted that if the target sentiment annotation data only includes sentence-level sentiment annotation data, it can be skipped and sentence-level sentiment control can be performed directly. Although the sentiment control granularity is relatively large compared to the first and third target sentiment annotation sequences, it can generate target synthesized speech data more quickly and efficiently.

[0149] Therefore, the target speech synthesis model of this disclosure supports both large-granularity emotion-controlled speech synthesis and small-granularity emotion-controlled speech synthesis. The former can efficiently obtain synthesized speech, while the latter can obtain more realistic and nuanced synthesized speech. The choice of emotion annotation data can be determined based on actual usage requirements, and this disclosure does not impose any restrictions on this.

[0150] In some optional implementations, the output data of the target speech synthesis model also includes: target alignment information, which is used to indicate the mapping relationship between the sentiment annotation data in the target sentiment annotation sequence and the speech segments in the target synthesized speech data. The target sentiment annotation sequence includes any one of the first target sentiment annotation sequence, the second target sentiment annotation sequence, and the third target sentiment annotation sequence.

[0151] Furthermore, target alignment information can be used to verify target synthesized speech data. For example, at least some target speech segments of the target synthesized speech data can be annotated with emotions or generate emotion scripts using a preset emotion processing model to obtain emotion prediction data. Based on this, it can be compared with the corresponding emotion annotation data to determine whether the corresponding target speech segment accurately and precisely expresses the preset emotion.

[0152] In summary, in this embodiment, the target speech synthesis model supports inputting sentiment-annotated data at the sentence level, as well as inputting sentiment-annotated data at the sentence level and sentiment script data at the segment level. It also supports direct input of sentiment-annotated sequences at the word level. It is compatible with various input data types and performs corresponding sentiment control for different input data, driving accurate and precise speech synthesis. In other words, the target speech synthesis model's compatibility with multiple sentiment control granularities allows it to revert to a sentence-level sentiment-controlled speech synthesis scheme, or be upgraded to a segment-level or word-level sentiment-controlled speech synthesis scheme, thus satisfying multiple speech synthesis needs with a single model.

[0153] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0154] Thirdly, embodiments of this disclosure provide a model training apparatus.

[0155] Figure 3 This is a block diagram of a model training apparatus provided in an embodiment of this disclosure. (Refer to...) Figure 3 The model training device 300 may include the following modules:

[0156] The base training module 301 is used to perform base training on the initial speech synthesis model based on the first training set data to obtain an intermediate speech synthesis model. The first training set data includes multiple first text data and first sentiment annotation data at the sentence granularity corresponding to each first text data. The initial speech synthesis model is used to expand the first sentiment annotation data into the first sentiment annotation sequence of the corresponding first text data according to the word granularity, and to perform speech synthesis of the corresponding first text data based on the first sentiment annotation sequence.

[0157] The fine-tuning training module 302 is used to fine-tune the intermediate speech synthesis model based on the second training set data to obtain the target speech synthesis model. The second training set data includes multiple second text data and first sentiment script data at the segment granularity corresponding to each second text data. The intermediate speech synthesis model is used to expand the first sentiment script data into the second sentiment annotation sequence of the corresponding second text data according to the word granularity, and to perform speech synthesis of the corresponding second text data based on the second sentiment annotation sequence.

[0158] It should be noted that the model training device can be used to execute the model training method of the embodiments of this disclosure. Please refer to the relevant description of the model training method, which will not be described in detail here.

[0159] Therefore, in this embodiment, by expanding the global sentiment annotation data into a sentiment annotation sequence in word-granularity (i.e., the first sentiment annotation data) during the base training process, the obtained intermediate speech synthesis model reserves a sentiment injection interface in word-granularity form, which provides an important foundation for realizing fine and accurate control and driving of speech synthesis based on word-granularity sentiment data. In the fine-tuning training, sentiment script data with a smaller sentiment annotation granularity than the global sentiment annotation data is introduced and expanded according to word granularity, so as to obtain a sentiment annotation sequence (i.e., the first sentiment annotation data) with relatively higher accuracy. Based on this sentiment annotation sequence, the sentiment injection interface reserved in the base training stage is used to drive speech synthesis. By iteratively updating the parameters, a target speech synthesis model that can accurately drive speech synthesis with word-granularity precision can be obtained.

[0160] Fourthly, embodiments of this disclosure provide a speech synthesis device.

[0161] Figure 4 This is a block diagram of a speech synthesis device provided in an embodiment of this disclosure. (Refer to...) Figure 4 The speech synthesis device 400 may include the following modules:

[0162] The synthesis module 401 is used to input the target text data and the target sentiment annotation data corresponding to the target text data into the target speech synthesis model to obtain the corresponding target synthesized speech data. The target sentiment annotation data includes at least one of the following: a first target sentiment annotation sequence at the word level, sentiment annotation data at the sentence level, and sentiment script data at the segment level.

[0163] The target speech synthesis model is obtained through the model training method described in any one of the embodiments of this disclosure.

[0164] Therefore, in this embodiment, the target speech synthesis model supports inputting sentiment-annotated data at the sentence level, as well as inputting sentiment-annotated data at the sentence level and sentiment script data at the segment level. It also supports directly inputting sentiment-annotated sequences at the word level. This allows it to be compatible with various input data types and to perform corresponding sentiment control for different input data, driving accurate and precise speech synthesis. In other words, the target speech synthesis model's compatibility with multiple sentiment control granularities allows it to revert to a sentence-level sentiment-controlled speech synthesis scheme and also upgrade to a segment-level or word-level sentiment-controlled speech synthesis scheme. A single model can meet multiple speech synthesis needs.

[0165] The modules in the aforementioned model training device and speech synthesis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0166] In addition, this disclosure also provides electronic devices and computer-readable storage media, all of which can be used to implement any of the model training methods or speech synthesis methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the relevant section on methods and will not be repeated here.

[0167] Figure 5 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0168] Reference Figure 5 This disclosure provides an electronic device, which includes: at least one processor 501, at least one memory 502, and one or more I / O interfaces 503; wherein, the memory 502 stores one or more computer programs that can be executed by at least one processor 501, and the one or more computer programs are executed by at least one processor 501 to enable at least one processor 501 to perform the above-described model training method or speech synthesis method.

[0169] Figure 6 A block diagram of another electronic device provided in an embodiment of this disclosure.

[0170] Reference Figure 6 This disclosure provides an electronic device that includes multiple processing cores 601 and an on-chip network 602. The multiple processing cores 601 are all connected to the on-chip network 602, which is used to exchange data between the multiple processing cores and external data.

[0171] One or more processing cores 601 store one or more instructions, and the one or more instructions are executed by one or more processing cores 601 to enable one or more processing cores 601 to perform the above-mentioned model training method or speech synthesis method.

[0172] The modules in the aforementioned electronic device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of the computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0173] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the aforementioned model training method or speech synthesis method. The computer-readable storage medium may be volatile or non-volatile.

[0174] This disclosure also provides a computer program product, including computer-readable code, which, when run in a processor of an electronic device, executes the aforementioned model training method or speech synthesis method.

[0175] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0176] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable program instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0177] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0178] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0179] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0180] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0181] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0182] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0183] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0184] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A model training method, characterized in that, include: The initial speech synthesis model is trained on the basis of the first training set data to obtain the intermediate speech synthesis model; wherein, the first training set data includes multiple first text data and first sentiment annotation data at the sentence granularity corresponding to each first text data, and the initial speech synthesis model is used to expand the first sentiment annotation data into the first sentiment annotation sequence of the corresponding first text data according to the word granularity, and to perform speech synthesis of the corresponding first text data based on the first sentiment annotation sequence. The intermediate speech synthesis model is fine-tuned and trained based on the second training set data to obtain the target speech synthesis model; wherein, the second training set data includes multiple second text data and first sentiment script data at the segment granularity corresponding to each second text data, and the intermediate speech synthesis model is used to expand the first sentiment script data into the second sentiment annotation sequence of the corresponding second text data according to the word granularity, and to perform speech synthesis of the corresponding second text data based on the second sentiment annotation sequence.

2. The method according to claim 1, characterized in that, The first training set data also includes first reference speech data corresponding to each of the first text data. The step of training the initial speech synthesis model on the first training set data to obtain an intermediate speech synthesis model includes: Multiple sets of the first text data and multiple sets of the first sentiment annotation data are input into the initial speech synthesis model. For any one set of the first text data, the initial speech synthesis model is used to perform: The first text data is segmented to obtain a first word sequence, which includes multiple first word elements. Expand the first sentiment annotation data corresponding to the first text data according to word granularity to obtain the initial sentiment annotation sequence corresponding to the first word sequence. The initial sentiment annotation sequence includes the initial sentiment annotation data corresponding to each first word element. The initial sentiment annotation data corresponding to at least a portion of the first word units in the initial sentiment annotation sequence are subjected to distribution differentiation processing to obtain the first sentiment annotation sequence; Furthermore, speech synthesis is performed based on the first sentiment annotation sequence and the corresponding first word sequence to obtain first synthesized speech data corresponding to the first text data; First audio loss data is determined based on at least one of the first synthesized speech data and the corresponding first reference speech data; The network parameters of the initial speech synthesis model are updated based on the first audio loss data.

3. The method according to claim 2, characterized in that, The step of expanding the first sentiment annotation data corresponding to the first text data according to word granularity to obtain the initial sentiment annotation sequence corresponding to the first word sequence includes: The first sentiment annotation data is determined as the initial sentiment annotation data for each of the first word elements; The initial sentiment annotation data of multiple first word elements are arranged according to the order of the first word elements in the first word sequence to obtain the initial sentiment annotation sequence corresponding to the first word sequence.

4. The method according to claim 2, characterized in that, After expanding the first sentiment annotation data corresponding to the first text data at the word granularity to obtain the initial sentiment annotation sequence corresponding to the first word sequence, the method further includes: The first emotion-annotated sequence is aligned with the corresponding first synthesized speech data to obtain alignment information, which is used to indicate the mapping relationship between the emotion-annotated data in the first emotion-annotated sequence and the speech segments in the corresponding first synthesized speech data.

5. The method according to claim 1, characterized in that, The second training set data also includes second reference speech data corresponding to each of the second text data, and the first sentiment script data includes at least one first segment annotation data; The step of fine-tuning the intermediate speech synthesis model based on the second training set data to obtain the target speech synthesis model includes: Multiple sets of second text data and multiple sets of first emotion script data are input into the intermediate speech synthesis model. For any set of second text data, the intermediate speech synthesis model is used to execute: The second text data is segmented to obtain a second word sequence, which includes multiple second word elements. Expand the first sentiment script data corresponding to the second text data at the word granularity to obtain the second sentiment annotation sequence corresponding to the second word sequence. The second sentiment annotation sequence includes sentiment annotation data corresponding to each second word element. And based on the second sentiment annotation sequence and the corresponding second word sequence, speech synthesis is performed to obtain second synthesized speech data corresponding to the second text data; Identify the speech segments in the second synthesized speech data that correspond to the annotation data of each first segment; Based on a preset emotion processing model, emotion prediction is performed on each of the speech segments to obtain segment emotion prediction data corresponding to each speech segment. The second audio loss data is determined based on at least one second synthesized speech data and the corresponding second reference speech data; Emotional loss data is determined based on at least one of the first segment annotation data and the corresponding segment sentiment prediction data; The intermediate speech synthesis model is updated with some network parameters based on the second audio loss data and the emotional loss data.

6. The method according to claim 5, characterized in that, The first segment annotation data is used to characterize the sentiment annotation data of the corresponding first word segment, and the first word segment includes multiple consecutive second words; The step of expanding the first sentiment script data corresponding to the second text data at the word granularity to obtain the second sentiment annotation sequence corresponding to the second word sequence includes: For any second word element, if the second word element is located in any first word element segment, the first segment annotation data corresponding to the first word element segment is used as the sentiment annotation data of the second word element in the second sentiment annotation sequence; If the second word element is not located in any first word element segment, the sentence-level sentiment annotation data or preset sentiment annotation data of the second text data shall be used as the sentiment annotation data of the second word element in the second sentiment annotation sequence.

7. The method according to claim 5, characterized in that, Before determining the second audio loss data based on at least one second synthesized speech data and the corresponding second reference speech data, the method further includes: The supervisory identifier of the second word element determines whether the second word element participates in the calculation of the second audio loss data and / or the emotional loss data; The supervision identifier is added to the second word sequence based on a preset supervision mask, and the supervision mask is of the same length as the second word sequence. When the supervision identifier is a valid value, the second word element corresponding to the supervision identifier is determined to participate in the calculation of the second audio loss data and / or the emotional loss data. When the supervision identifier is an invalid value, the second word element corresponding to the supervision identifier is determined not to participate in the calculation of the second audio loss data and / or the emotional loss data.

8. The method according to any one of claims 5-7, characterized in that, The step of determining the sentiment loss data based on at least one of the first segment annotation data and the corresponding segment sentiment prediction data includes: For any of the aforementioned segment sentiment prediction data, an initial sentiment loss value is calculated based on the segment sentiment prediction data and the corresponding first segment annotation data; Based on the confidence levels of the multiple sentiment prediction data segments, the loss weight corresponding to each sentiment prediction data segment is determined; The emotional loss data is obtained by performing a weighted calculation based on the loss weight and the initial emotional loss value.

9. The method according to any one of claims 1 to 4, characterized in that, Before fine-tuning the intermediate speech synthesis model based on the second training set data to obtain the target speech synthesis model, the method further includes: Obtain a text dataset, which includes multiple third-party text data; Generate second sentiment script data at the fragment granularity corresponding to each of the third text data; Multiple third text data and multiple second emotion script data are input into the intermediate speech synthesis model to perform speech synthesis, thereby obtaining third synthesized speech data corresponding to the third text data; The second training set data is obtained based on multiple sets of third text data, multiple sets of second emotion script data, and multiple sets of third synthesized speech data.

10. The method according to claim 9, characterized in that, The generation of second sentiment script data at the fragment granularity corresponding to each of the third text data includes: For any of the aforementioned third text data, the third text data is segmented into words to obtain a third word sequence; The third word sequence is sampled to obtain at least one second word segment; Sentiment annotation is performed on each of the second word segments to obtain the second sentiment script data corresponding to the third text data; The second sentiment script data includes second segment annotation data corresponding to each second word segment.

11. The method according to claim 9, characterized in that, The step of inputting multiple sets of third text data and multiple sets of second emotion script data into the intermediate speech synthesis model for speech synthesis to obtain third synthesized speech data corresponding to the third text data includes: Multiple sets of the third text data and multiple sets of the second emotion script data are input into the intermediate speech synthesis model. For any one of the third text data, the intermediate speech synthesis model is used to execute: The third text data is segmented to obtain a third word sequence, which includes multiple third word elements; Expand the second sentiment script data corresponding to the third text data at the word granularity to obtain the third sentiment annotation sequence corresponding to the third word sequence. The third sentiment annotation sequence includes sentiment annotation data corresponding to each third word element. Speech synthesis is performed based on the third emotion annotation sequence and the corresponding third word sequence to obtain third synthesized speech data corresponding to the third text data.

12. The method according to claim 11, characterized in that, The second sentiment script data includes at least one second segment annotation data, which is used to characterize the sentiment annotation data of the corresponding second word segment, and the second word segment includes multiple consecutive third words; The step of expanding the second sentiment script data corresponding to the third text data at the word granularity to obtain the third sentiment annotation sequence corresponding to the third word sequence includes: For any of the third word elements, if the third word element is located in any second word element segment, the second segment annotation data corresponding to the second word element segment is used as the sentiment annotation data of the third word element in the third sentiment annotation sequence; If the third word element is not located in any second word element segment, the sentence-level sentiment annotation data or preset sentiment annotation data of the third text data shall be used as the sentiment annotation data of the third word element in the third sentiment annotation sequence.

13. The method according to claim 9, characterized in that, Before inputting multiple sets of third text data and multiple sets of second emotion script data into the intermediate speech synthesis model for speech synthesis to obtain third synthesized speech data corresponding to the third text data, the method further includes: Based on preset quality indicators, delete a portion of the third text data and the corresponding second emotion script data, and / or correct a portion of the second emotion script data of the third text data.

14. The method according to claim 13, characterized in that, The step of deleting a portion of the third text data and the corresponding second emotional script data according to a preset quality index, and / or correcting a portion of the second emotional script data of the third text data, includes: Based on a preset sentiment annotation model, a prediction script data and a confidence level of the prediction script data are generated for each of the third text data. Delete the third text data and the corresponding second sentiment script data whose confidence level is less than the preset confidence threshold; and / or, For any of the third text data, a consistency check is performed on the second sentiment script data and prediction script data corresponding to the third text data to determine the consistency index; Based on the consistency index, at least one third text data is selected from multiple third text data, and the second sentiment script data corresponding to the at least one third text data is corrected to obtain the corrected second sentiment script data corresponding to the at least one third text data.

15. A speech synthesis method, characterized in that, include: The target text data and the target sentiment annotation data corresponding to the target text data are input into the target speech synthesis model to obtain the corresponding target synthesized speech data. The target sentiment annotation data includes at least one of the following: a first target sentiment annotation sequence at the word level, sentiment annotation data at the sentence level, and sentiment script data at the segment level. The target speech synthesis model is obtained by the model training method as described in any one of claims 1-14.

16. The method according to claim 15, characterized in that, When the target emotion annotation data includes the first target emotion annotation sequence, speech synthesis is performed based on the first target emotion annotation sequence; When the target sentiment annotation data includes sentence-level sentiment annotation data, the sentence-level sentiment annotation data is expanded into a second target sentiment annotation sequence with word-level corresponding to the target word sequence, and speech synthesis is performed based on the second target sentiment annotation sequence, wherein the target word sequence is obtained by segmenting the target text data; When the target sentiment annotation data includes sentence-level sentiment annotation data and segment-level sentiment script data, an expansion process is performed based on the sentence-level sentiment annotation data and segment-level sentiment script data to obtain a third target sentiment annotation sequence at the word level corresponding to the target word sequence, and speech synthesis is performed based on the third target sentiment annotation sequence.

17. The method according to claim 16, characterized in that, The output data of the target speech synthesis model further includes: target alignment information, which is used to indicate the mapping relationship between the sentiment annotation data in the target sentiment annotation sequence and the speech segments in the target synthesized speech data. The target sentiment annotation sequence includes any one of the first target sentiment annotation sequence, the second target sentiment annotation sequence, and the third target sentiment annotation sequence.

18. A model training device, characterized in that, include: The base training module is used to perform base training on the initial speech synthesis model based on the first training set data to obtain the intermediate speech synthesis model; wherein, the first training set data includes multiple first text data and first sentiment annotation data at the sentence granularity corresponding to each first text data, and the initial speech synthesis model is used to expand the first sentiment annotation data into the first sentiment annotation sequence of the corresponding first text data according to the word granularity, and perform speech synthesis of the corresponding first text data based on the first sentiment annotation sequence; The fine-tuning training module is used to fine-tune the intermediate speech synthesis model based on the second training set data to obtain the target speech synthesis model; wherein, the second training set data includes multiple second text data and first sentiment script data at the segment granularity corresponding to each second text data, and the intermediate speech synthesis model is used to expand the first sentiment script data into the second sentiment annotation sequence of the corresponding second text data according to the word granularity, and perform speech synthesis of the corresponding second text data based on the second sentiment annotation sequence.

19. A speech synthesis device, characterized in that, include: The synthesis module is used to input target text data and target sentiment annotation data corresponding to the target text data into the target speech synthesis model to obtain the corresponding target synthesized speech data. The target sentiment annotation data includes at least one of word-level first target sentiment annotation sequence, sentence-level sentiment annotation data, and segment-level sentiment script data. The target speech synthesis model is obtained by the model training method as described in any one of claims 1-14.

20. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the model training method as described in any one of claims 1-14, or the speech synthesis method as described in any one of claims 15-17.

21. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the model training method as described in any one of claims 1-14, or the speech synthesis method as described in any one of claims 15-17.

22. A computer program product, characterized in that, It includes computer-readable code, which, when executed in a processor of an electronic device, enables the processor in the electronic device to perform a model training method as described in any one of claims 1-14, or a speech synthesis method as described in any one of claims 15-17.

Images