Cross-language emotional speech synthesis method, device and equipment based on ByT5 and medium

Abstract

The invention belongs to the technical field of intelligent speech synthesis, and discloses a ByT5-based cross-language emotional speech synthesis method, device and equipment and a medium, and the method comprises the steps: obtaining a part-of-speech information embedding vector, a syntactic information embedding vector and a text feature vector of an input text; performing feature fusion to obtain an emotion text implicit vector and a speaker text implicit vector; according to the emotion text implicit vector and the speaker text implicit vector, respectively generating an emotion speech MEL spectrum and a speaker speech MEL spectrum, generating a target MEL spectrum by using the emotion speech MEL spectrum and the speaker speech MEL spectrum, and discriminating the target MEL spectrum by using a discriminator; and respectively performing adversarial training on a generator and a discriminator of the cross-language emotional speech synthesis model by utilizing the generation loss function and the discrimination loss function until the model converges. The problem that a token-based model is limited by a dictionary is solved, the robustness of the model is improved, the quality and naturalness of the synthesized speech are effectively improved, and the speaker characteristics and emotion characteristics of the speech are controllable.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent speech synthesis, and in particular to a cross-language emotional speech synthesis method, device, equipment and medium based on ByT5. Background Art

[0002] The cross-language speech synthesis task involves converting text in one language into natural and fluent speech in another language. This task has important application prospects in multilingual society and international applications. The speech synthesis function with emotions and specific speaker characteristics expanded on the cross-language function can further improve the user experience of the voice interaction system and provide a wider range of services.

[0003] Currently, most TTS systems need to work based on tokens during the text processing stage. The defects of this method are: the model can only recognize and process characters within the vocabulary range (that is, there are corresponding tokens). For unknown characters, the model is often unable to process them. In addition, the speech synthesized by existing cross-language speech synthesis methods is often not good and there are problems such as semantic loss. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a cross-language emotional speech synthesis method, device, equipment and medium based on ByT5 to solve the problems of limited recognition range of existing speech models and poor synthesis speech effect.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a cross-language emotional speech synthesis method based on ByT5, which adopts the following technical solution, including:

[0006] Obtain the part-of-speech information embedding vector and syntactic information embedding vector of the input text, use BYT5 to convert the bit sequence of the input text, and obtain the text feature vector;

[0007] Obtaining an emotion ID embedding vector and a speaker ID embedding vector according to the emotion ID information and the speaker ID information respectively, performing feature fusion on the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the emotion ID embedding vector respectively, obtaining an emotion text latent vector, performing feature fusion on the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the speaker ID feature vector respectively, obtaining a speaker text latent vector;

[0008] Generate an emotional speech MEL spectrum and a speaker speech MEL spectrum according to the emotional text latent vector and the speaker text latent vector respectively, generate a target MEL spectrum using the emotional speech MEL spectrum and the speaker speech MEL spectrum, and use a discriminator to discriminate the target MEL spectrum;

[0009] The emotional speech MEL spectrum and the target language speech MEL spectrum with the speaker ID blurred are used to obtain the emotional speech loss function, the speaker speech MEL spectrum and the MEL spectrum of the target language speech with the emotional ID blurred are used to obtain the speaker speech loss function, the generation loss function is obtained according to the speaker speech loss function, the emotional speech loss function, the generator loss function and the L2 loss function, the discriminant loss function is obtained, and the generator and the discriminant of the cross-language emotional speech synthesis model are respectively trained adversarially using the generation loss function and the discriminant loss function until the model converges to obtain the target cross-language emotional speech synthesis model;

[0010] The text to be synthesized is received, and the text to be synthesized is input into the target cross-language emotional speech synthesis model to obtain the target speech.

[0011] Furthermore, the step of obtaining the part-of-speech information embedding vector of the input text includes:

[0012] Preprocessing the input text, using a part-of-speech tagging tool to perform part-of-speech tagging on each word in the preprocessed text, and obtaining a part-of-speech tagging table;

[0013] Each word and part-of-speech combination in the part-of-speech tagging table is represented by a vector according to a specified vector dimension to obtain the part-of-speech information embedding vector.

[0014] Further, the step of obtaining the syntactic information embedding vector of the input text includes:

[0015] Preprocessing the input text, performing syntactic analysis on the preprocessed text using a syntactic analysis tool, and obtaining a syntactic tree;

[0016] Performing feature extraction on the syntax tree to obtain syntax features;

[0017] Each syntactic feature is represented by a vector according to a specified vector dimension to obtain the syntactic information embedding vector.

[0018] Furthermore, the feature fusion of the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the emotion ID embedding vector is performed to obtain the emotion text latent vector, including:

[0019] Preprocessing the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the sentiment ID embedding vector to keep the dimensions of all embedding vectors consistent;

[0020] The preprocessed part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the emotion ID embedding vector are concatenated to obtain a first emotion speech long vector;

[0021] Inputting the first emotional speech long vector into a fully connected layer for linear operation to obtain a second emotional speech long vector;

[0022] The second emotional speech long vector is input into the latent function layer, and nonlinear transformation is performed to obtain the emotional text latent vector.

[0023] Furthermore, the feature fusion of the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the speaker ID feature vector to obtain the speaker text latent vector includes:

[0024] Preprocessing the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the speaker ID embedding vector to make the dimensions of all embedding vectors consistent;

[0025] Concatenate the preprocessed part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the speaker ID embedding vector to obtain a first speaker speech long vector;

[0026] Inputting the first speaker's speech long vector into a fully connected layer for linear operation to obtain a second speaker's speech long vector;

[0027] The second speaker's speech long vector is input into the latent function layer and nonlinearly transformed to obtain the speaker's text latent vector.

[0028] Furthermore, the method of obtaining the emotional speech loss function by using the emotional speech MEL spectrum and the target language speech MEL spectrum with the speaker ID blurred includes:

[0029] Acquire a first target language speech corresponding to the input text, and use a translation function to translate the speech formant of the first target language speech to obtain a second target language speech, wherein the speaker ID of the second target language speech is blurred;

[0030] The MEL spectrum and the emotional speech MEL spectrum of the second target language speech are obtained, and the emotional speech loss function is obtained according to the MEL spectrum and the emotional speech MEL spectrum of the second target language speech.

[0031] Furthermore, the method of obtaining the speaker speech loss function by using the speaker speech MEL spectrum and the MEL spectrum of the target language speech whose emotion ID is blurred includes:

[0032] Acquire a first target language speech corresponding to the input text, and use an elimination function to eliminate the emotion information of the first target language speech to obtain a third target language speech, wherein the emotion ID of the third target language speech is blurred;

[0033] The MEL spectrum of the third target language speech and the MEL spectrum of the speaker speech are obtained, and the speaker speech loss function is obtained according to the MEL spectrum of the third target language speech and the MEL spectrum of the speaker speech.

[0034] In order to solve the above problems, a cross-language emotional speech synthesis device based on ByT5 is also provided, which adopts the cross-language emotional speech synthesis method based on ByT5, including:

[0035] The first acquisition module is used to obtain the part-of-speech information embedding vector and the syntactic information embedding vector of the input text, and convert the bit sequence of the input text using BYT5 to obtain the text feature vector;

[0036] The second acquisition module is used to obtain the emotion ID embedding vector and the speaker ID embedding vector according to the emotion ID information and the speaker ID information, respectively, and to fuse the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the emotion ID embedding vector to obtain the emotion text latent vector, and to fuse the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the speaker ID feature vector to obtain the speaker text latent vector;

[0037] A generation module, used to generate an emotional speech MEL spectrum and a speaker speech MEL spectrum according to an emotional text latent vector and a speaker text latent vector, generate a target MEL spectrum using the emotional speech MEL spectrum and the speaker speech MEL spectrum, and discriminate the target MEL spectrum using a discriminator;

[0038] A training module, for obtaining an emotional speech loss function using the emotional speech MEL spectrum and the MEL spectrum of the target language speech with the speaker ID blurred, obtaining a speaker speech loss function using the speaker speech MEL spectrum and the MEL spectrum of the target language speech with the emotional ID blurred, obtaining a speaker speech loss function, an emotional speech loss function, a generator loss function and an L2 loss function, and obtaining a generation loss function based on the speaker speech loss function, the emotional speech loss function, the generator loss function and the L2 loss function, obtaining a discriminant loss function, and using the generation loss function and the discriminant loss function to perform adversarial training on the generator and the discriminator of the cross-language emotional speech synthesis model respectively until the model converges to obtain a target cross-language emotional speech synthesis model;

[0039] The synthesis module is used to receive the text to be synthesized, input the text to be synthesized into the target cross-language emotional speech synthesis model, and obtain the target speech.

[0040] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the cross-language emotional speech synthesis method based on ByT5 as described above are implemented.

[0041] In order to solve the above technical problems, an embodiment of the present application also provides a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the steps of the cross-language emotional speech synthesis method based on ByT5 as described above are implemented.

[0042] Compared with the prior art, the embodiments of the present application mainly have the following technical effects: by using BYT5 to convert the bit sequence of the input text and obtain the text feature vector, the text is stored in the computer device in the form of bytes, and any text and vocabulary phrases can be represented by bytes or byte sequences, making full use of the byte encoding of the ByT5 architecture to represent the advantages in solving multilingual and new word problems, solving the problem of the token-based model being restricted by the dictionary, and improving the robustness of the model. By integrating feature information such as part of speech, syntactic structure, speaker, emotion, etc., the quality and naturalness of the synthesized speech are effectively improved, the speaker characteristics and emotional characteristics of the speech are controllable, and the emotional and speaker characteristics obtained after decoding are used to enable the model to learn these features more effectively. By conducting adversarial training on the model, the interference of information noise in the modeling process is reduced, and the quality of the synthesized speech is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A flowchart of an embodiment of a cross-language emotional speech synthesis method based on ByT5 according to the present application;

[0045] Figure 2 yes Figure 1 A flowchart of a specific implementation method of obtaining a part-of-speech information embedding vector in step 100;

[0046] Figure 3 yes Figure 1 A flowchart of a specific implementation of step 100 of obtaining a syntactic information embedding vector;

[0047] Figure 4 yes Figure 1 A flowchart of a specific implementation method of obtaining a latent vector of a sentiment text in step 200;

[0048] Figure 5 yes Figure 1 A flowchart of a specific implementation of step 200 of the speaker text latent vector;

[0049] Figure 6 yes Figure 1 Step 400 is a flowchart of a specific implementation method of obtaining the emotional speech loss function;

[0050] Figure 7 yes Figure 1 A flowchart of a specific implementation method of obtaining a speaker speech loss function in step 400;

[0051] Figure 8 It is a flowchart of a cross-language emotional speech synthesis method based on ByT5 according to the present application;

[0052] Fig. 9 It is a structural schematic diagram of an embodiment of a cross-language emotional speech synthesis device based on ByT5 according to the present application;

[0053] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the accompanying drawings in the invention to clearly and completely describe the technical solutions in the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the scope of protection of the present invention.

[0055] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used in the specification of the application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0057] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0058] The cross-language emotional speech synthesis method based on ByT5 provided in the embodiment of the present application can be applied to a computer device with the ability to run instructions or programs. Among them, the computer device can be but is not limited to various servers, workstations, personal computers, laptops, etc. Running on different computing devices is only a difference in the execution subject of the scheme. People in this field can foresee that running in different computing devices can produce the same technical effects. The present invention is described in detail below through specific embodiments.

[0059] The purpose of the embodiments of the present application is to propose a cross-language emotional speech synthesis method, device, equipment and medium based on ByT5 to solve the problems of limited recognition range of existing speech models and poor synthesis speech effect.

[0060] TTS (Text-to-Speech) text model is a technical model that can convert input text information into natural and fluent speech output. Common TTS (Text-to-Speech) models include VITS (Variational Inference with Adversarial Learning for End-to-End Text-to-Speech) model, YourTTS model, PhemeTTS model, StyleTTS2 model, etc. These models need to work based on tokens in the text processing stage. The technical problems of these technical models are: the models can only recognize and process characters within the vocabulary range (that is, there are corresponding tokens). They are often unable to process unknown words, and the speech synthesis effect is not satisfactory.

[0061] In order to solve the above technical problems, the present application embodiment provides a cross-language emotional speech synthesis method based on ByT5, which adopts the following technical solutions: Figure 1 , Figure 1 A flowchart of an embodiment of a cross-language emotional speech synthesis method based on ByT5 according to the present application includes:

[0062] S100, obtaining a part-of-speech information embedding vector and a syntactic information embedding vector of an input text, converting a bit sequence of the input text using BYT5, and obtaining a text feature vector.

[0063] ByT5 (Byte-level T5) is a language model based on the T5 (Text-To-Text Transfer Transformer) architecture. It inherits the basic architecture of T5 and adopts the encoder-decoder structure of Transformer. This structure performs well in natural language processing tasks and can effectively capture the contextual information and semantic relationships of the text. By pre-training at the byte level, the model can learn the byte-level representation of the text, so as to better understand and generate text. A large amount of text data is used in the pre-training process, so that the model can learn the common features and patterns of different languages.

[0064] In this embodiment, when obtaining a text feature vector, the input text is first converted into a bit sequence.

[0065] Specifically, the input text consists of characters, and each character has a corresponding encoding representation in the computer. Common encoding methods include ASCII code, Unicode, etc. These encodings map characters to specific digital values, which can then be further converted into binary (bit sequence) representations.

[0066] You can choose ASCII encoding, and convert each character into its corresponding digital code value according to the selected encoding method. For example, in ASCII code, the character "A" corresponds to the number 65. Next, convert the digital code value of each character into a binary representation. For example, the binary representation of the number 65 is 1000001. Then, concatenate the binary representations of each character in turn to get the bit sequence of the input text.

[0067] For example, for the text "Hello", assuming it is encoded in ASCII code: "H" corresponds to the number 72, which is 1001000 in binary. "e" corresponds to the number 101, which is 1100101 in binary. "l" corresponds to the number 108, which is 1101100 in binary. "l" again corresponds to the number 108, which is 1101100 in binary. "o" corresponds to the number 111, which is 1101111 in binary. Concatenating these binary sequences, the bit sequence of "Hello" is 1001000110010111011001101101111.

[0068] In this embodiment, the input text is converted into byte form and stored in a computer device. Any text and vocabulary phrase can be represented by bytes or byte sequences, so the ByT5 architecture that uses byte encoding to represent characters shows outstanding superiority in solving multilingual and new word problems. The cross-language emotional speech synthesis using the ByT5 architecture in the embodiment of the present application is more robust and performs better on word-level tasks that are sensitive to spelling and pronunciation.

[0069] In some preferred embodiments, Figure 2 , Figure 2 yes Figure 1 S100 is a flowchart of a specific implementation method for obtaining a part-of-speech information embedding vector; S100 includes: S110, preprocessing the input text, using a part-of-speech tagging tool to perform part-of-speech tagging on each word in the preprocessed text, and obtaining a part-of-speech tagging table; S120, performing vector representation on each word and part-of-speech combination in the part-of-speech tagging table according to a specified vector dimension, and obtaining a part-of-speech information embedding vector.

[0070] In this embodiment, the input text is first preprocessed, including operations such as removing special characters, normalizing punctuation marks, and unifying upper and lower case characters.

[0071] Use the part-of-speech tagging tool to tag each word in the text. The tagging results may include nouns, verbs, adjectives, adverbs, etc.

[0072] In this embodiment, a part-of-speech tagging table is constructed using all different words and part-of-speech combinations that appear in the input text. In addition, a unique index may be assigned to each word and part-of-speech combination.

[0073] like Figure 8 , Figure 8 It is a flowchart of a cross-language emotional speech synthesis method based on ByT5 according to the present application;

[0074] like Figure 8, use the part-of-speech tagging tool and the syntactic analysis tool to obtain the part-of-speech information embedding vector and the syntactic information embedding vector of the input text, and use BYT5 to convert the bit sequence of the input text to obtain the text feature vector;

[0075] Obtain the emotion ID embedding vector and speaker ID embedding vector according to the emotion ID information and speaker ID information respectively, perform feature fusion on the part-of-speech information embedding vector, syntactic information embedding vector, text feature vector and emotion ID embedding vector respectively, obtain the emotion text latent vector, perform feature fusion on the part-of-speech information embedding vector, syntactic information embedding vector, text feature vector and speaker ID feature vector respectively, obtain the speaker text latent vector;

[0076] Generate emotional speech MEL spectrum and speaker speech MEL spectrum according to emotional text latent vector and speaker text latent vector respectively, generate target MEL spectrum by using emotional speech MEL spectrum and speaker speech MEL spectrum, and discriminate the target MEL spectrum by using discriminator;

[0077] The emotional speech loss function is obtained by using the emotional speech MEL spectrum and the MEL spectrum of the target language speech with the speaker ID blurred. The speaker speech loss function is obtained by using the speaker speech MEL spectrum and the MEL spectrum of the target language speech with the emotional ID blurred. The generation loss function is obtained according to the speaker speech loss function, the emotional speech loss function, the generator loss function and the L2 loss function. The generation loss function and the discriminator loss function are used to perform adversarial training on the generator and the discriminator of the cross-language emotional speech synthesis model respectively until the model converges to obtain the target cross-language emotional speech synthesis model.

[0078] In this embodiment, by obtaining the part-of-speech information embedding vector and combining the part-of-speech embedding vector with other features, a richer input feature can be provided for the natural language processing model, thereby improving the performance of the model. For example, in the named entity recognition task, part-of-speech information can help the model better identify the type of entity. It can also help the model better generalize to different text fields and language variants. Since the part-of-speech has a certain stability in different texts, the introduction of part-of-speech information can make the model more robust when facing new input texts.

[0079] In some optional embodiments, such as Figure 3 , Figure 3 yes Figure 1S100 is a flowchart of a specific implementation method for obtaining a syntactic information embedding vector, S100 includes: S130, preprocessing the input text, performing syntactic analysis on the preprocessed text using a syntactic analysis tool, and obtaining a syntactic tree; S140, performing feature extraction on the syntactic tree to obtain syntactic features; S150, vectorizing each syntactic feature according to a specified vector dimension to obtain a syntactic information embedding vector.

[0080] In this embodiment, the input text is firstly cleaned up to remove unnecessary noise, such as special characters, garbled characters, etc. The text is syntactically analyzed using a syntactic analysis tool. The analysis results will give the component structure of the sentence, such as subject, predicate, object, attributive, adverbial, complement, etc., as well as the dependency relationship between words.

[0081] In this embodiment, a syntactic tree representation of a sentence is constructed based on the results of syntactic analysis. The syntactic tree can clearly show the structural hierarchy of a sentence and the relationship between words. Useful features such as node type, node depth, subtree structure, etc. are extracted from the syntactic tree.

[0082] During the training process, the weights of the vectors are continuously adjusted so that sentences with similar syntactic structures are closer in the vector space.

[0083] In this embodiment, by obtaining the syntactic information embedding vector, additional and valuable features are provided for the cross-language emotional speech synthesis based on ByT5. Combined with the part-of-speech embedding vector feature, the expressive power of the cross-language emotional speech synthesis model based on ByT5 can be significantly enhanced. For example, in text classification tasks, the syntactic information embedding vector can help the model better distinguish different types of text structures, thereby improving classification accuracy. It can also achieve better performance on different data sets and tasks, reducing excessive reliance on data in specific fields.

[0084] S200, respectively obtaining the emotion ID embedding vector and the speaker ID embedding vector according to the emotion ID information and the speaker ID information, respectively performing feature fusion on the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the emotion ID embedding vector to obtain the emotion text latent vector, respectively performing feature fusion on the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the speaker ID feature vector to obtain the speaker text latent vector.

[0085] In some optional embodiments, such as Figure 4 , Figure 4 yes Figure 1 S200 is a flowchart of a specific implementation method of obtaining a latent vector of a sentiment text, wherein S200 includes:

[0086] S210, preprocess the part-of-speech information embedding vector, syntactic information embedding vector, text feature vector, and emotion ID embedding vector to keep the dimensions of all embedding vectors consistent; S220, concatenate the preprocessed part-of-speech information embedding vector, syntactic information embedding vector, text feature vector, and emotion ID embedding vector to obtain a first emotion speech long vector; S230, input the first emotion speech long vector into the fully connected layer for linear operation to obtain a second emotion speech long vector; S240, input the second emotion speech long vector into the latent function layer for nonlinear transformation to obtain an emotion text latent vector.

[0087] In this embodiment, the acquired part-of-speech information embedding vector, syntactic information embedding vector, text feature vector, and emotion ID embedding vector are preprocessed to ensure that the dimensions and data types of each feature are consistent. Specific preprocessing includes removing outliers in the vector, processing missing values, and performing normalization. After preprocessing, these feature vectors are spliced ​​together to form a long vector as the input of the fully connected layer. To initialize the weight matrix and bias term of the fully connected layer, common initialization methods such as Xavier initialization or He initialization can be used. The first emotional speech long vector obtained after splicing is input into the fully connected layer. The fully connected layer performs a linear operation, that is, the feature vector is multiplied by the weight matrix, and then the bias term is added to obtain a new vector, which is the second emotional speech long vector.

[0088] In this embodiment, through the above steps, the fully connected layer can effectively fuse the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the sentiment ID embedding vector to obtain a more powerful latent vector and provide better feature representation for subsequent tasks.

[0089] In some optional embodiments, such as Figure 5 , Figure 5 yes Figure 1 A flowchart of a specific implementation of the speaker text latent vector in S200 includes:

[0090] S250, preprocessing the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the speaker ID embedding vector to keep the dimensions of all the embedding vectors consistent; S260, concatenating the preprocessed part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the speaker ID embedding vector to obtain the first speaker's speech long vector; S270, inputting the first speaker's speech long vector into the fully connected layer for linear operation to obtain the second speaker's speech long vector; S280, inputting the second speaker's speech long vector into the latent function layer for nonlinear transformation to obtain the speaker's text latent vector.

[0091] In this embodiment, the acquired part-of-speech information embedding vector, syntactic information embedding vector, text feature vector, and speaker ID embedding vector are preprocessed to ensure that the dimensions and data types of each feature are consistent. The specific preprocessing includes removing outliers in the vector, processing missing values, and performing normalization. After preprocessing, these feature vectors are spliced ​​together to form a long vector as the input of the fully connected layer. To initialize the weight matrix and bias term of the fully connected layer, common initialization methods such as Xavier initialization or He initialization can be used. The long vector of the first speaker's speech obtained after splicing is input into the fully connected layer. The fully connected layer performs a linear operation, that is, the feature vector is multiplied by the weight matrix, and then the bias term is added to obtain a new vector, that is, the long vector of the second speaker's speech.

[0092] In this embodiment, through the above steps, the fully connected layer can effectively fuse the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the speaker ID embedding vector to obtain a more powerful latent vector and provide better feature representation for subsequent tasks.

[0093] S300, generating an emotional speech MEL spectrum and a speaker speech MEL spectrum according to the emotional text latent vector and the speaker text latent vector respectively, generating a target MEL spectrum using the emotional speech MEL spectrum and the speaker speech MEL spectrum, and using a discriminator to discriminate the target MEL spectrum.

[0094] like Figure 8 , the emotional text latent vector and the speaker text latent vector are input into the corresponding generator of the cross-language emotional speech synthesis model in the embodiment of the present application, and the emotional speech MEL spectrum and the speaker speech MEL spectrum are generated respectively. Then, the emotional speech MEL spectrum and the speaker speech MEL spectrum generated above are passed through the decoder of the model to generate the target MEL spectrum of the target language, and the generated target MEL spectrum is compared with the real MEL spectrum for discrimination and classification.

[0095] S400, using the emotional speech MEL spectrum and the MEL spectrum of the target language speech with the speaker ID blurred to obtain the emotional speech loss function, using the speaker speech MEL spectrum and the MEL spectrum of the target language speech with the emotional ID blurred to obtain the speaker speech loss function, according to the speaker speech loss function, the emotional speech loss function, the generator loss function and the L2 loss function to obtain the generation loss function, obtain the discriminant loss function, use the generation loss function and the discriminant loss function to perform adversarial training on the generator and the discriminator of the cross-language emotional speech synthesis model respectively, until the model converges, and obtain the target cross-language emotional speech synthesis model.

[0096] In some optional embodiments, such as Figure 6 , Figure 6 yes Figure 1 S400 is a flowchart of a specific implementation method of obtaining an emotional speech loss function; S400 includes:

[0097] S410, obtaining a first target language speech corresponding to the input text, using a translation function to translate the speech resonance peaks of the first target language speech, to obtain a second target language speech, wherein the speaker ID of the second target language speech is blurred; S420, obtaining a MEL spectrum and an emotional speech MEL spectrum of the second target language speech, and obtaining an emotional speech loss function based on the MEL spectrum and the emotional speech MEL spectrum of the second target language speech.

[0098] In this embodiment, an "information fuzziness" implementation method is used to better separate speaker information and emotion information in speech.

[0099] Since the timbre of a speaker is closely related to the formant of speech, in this embodiment, the speaker identity information in the speech signal is disturbed by interfering with the formant of speech, while the emotional expression is expected to remain unchanged.

[0100] In this embodiment, a translation function f is used p , the resonance peak is moved within a random range, so that the speaker information in the original speech is eliminated. Finally, the target language speech with blurred speaker information and the corresponding Mel spectrum are obtained, that is, the MEL spectrum of the second target language speech and the MEL spectrum of the emotional speech.

[0101] In this implementation, by adopting the "information fuzziness" implementation method to obtain emotional information and obtain the emotional speech loss function, the process of generating specified emotions based on the ByT5 cross-language emotional speech synthesis model is made more controllable.

[0102] In some optional embodiments, such as Figure 7 , Figure 7 yes Figure 1 S400 is a flowchart of a specific implementation method for obtaining a speaker speech loss function; S400 includes: S430, obtaining a first target language speech corresponding to an input text, eliminating the emotion information of the first target language speech by using an elimination function, obtaining a third target language speech, and the emotion ID of the third target language speech is blurred; S440, obtaining a MEL spectrum of the third target language speech and a speaker speech MEL spectrum, and obtaining a speaker speech loss function based on the MEL spectrum of the third target language speech and the speaker speech MEL spectrum.

[0103] In this embodiment, an "information fuzziness" implementation method is used to better separate speaker information and emotion information in speech.

[0104] In this implementation, based on the close relationship between emotional expression and pitch and pitch trend, the pitch information is disturbed by random translation and averaging.

[0105] Specifically, in this embodiment, the pitch of the speech is first shifted by a random amplitude. Since the pitch variation trend in speech of different emotion categories is still retained after the shift, all pitch values ​​are further monotonized to the average pitch. In the process of eliminating emotional information, we use the function f h It indicates that the target language speech with blurred emotional information and the corresponding Mel spectrum are obtained, that is, the MEL spectrum of the third target language speech and the MEL spectrum of the speaker's speech.

[0106] In this implementation, the speaker information is obtained by adopting the "information fuzziness" implementation method, and the corresponding speaker speech loss function is obtained, so that the process of generating specified emotions based on the ByT5 cross-language emotional speech synthesis model is more controllable.

[0107] S500: receiving a text to be synthesized, and inputting the text to be synthesized into the target cross-language emotional speech synthesis model to obtain a target speech.

[0108] In an embodiment of the present application, when training a cross-language emotional speech synthesis model based on ByT5, an adversarial training strategy is adopted, and the generated target speech Mel spectrum and the real Mel spectrum are input into a discriminator to distinguish true from false.

[0109] Assume that the emotional speech loss function and the speaker speech loss function are L emotion , L speaker , the loss function of the Mel spectrum generator is L Mel , the L2 loss function between the generated target Mel spectrum and the real Mel spectrum is L rec , then the loss function L of the training discriminator is D for::

[0110]

[0111] The loss function L for training the generator G for:

[0112]

[0113] in, α, β, γ, and δ are hyperparameters for adjusting the weights of each loss and can be adjusted according to the actual training situation.

[0114] In order to solve the above technical problems, the present application embodiment also provides a cross-language emotional speech synthesis device 600 based on ByT5, such as Fig. 9 , Fig. 91 is a schematic diagram of a structure of an embodiment of a cross-language emotional speech synthesis device 600 based on ByT5 according to the present application; the cross-language emotional speech synthesis method based on ByT5 of the first aspect is adopted, comprising:

[0115] The first acquisition module 601 is used to acquire the part-of-speech information embedding vector and the syntactic information embedding vector of the input text, and convert the bit sequence of the input text using BYT5 to acquire the text feature vector;

[0116] The second acquisition module 602 is used to obtain the emotion ID embedding vector and the speaker ID embedding vector according to the emotion ID information and the speaker ID information, respectively, to fuse the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the emotion ID embedding vector to obtain the emotion text latent vector, and to fuse the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the speaker ID feature vector to obtain the speaker text latent vector;

[0117] A generation module 603 is used to generate an emotional speech MEL spectrum and a speaker speech MEL spectrum according to the emotional text latent vector and the speaker text latent vector, generate a target MEL spectrum using the emotional speech MEL spectrum and the speaker speech MEL spectrum, and discriminate the target MEL spectrum using a discriminator;

[0118] The training module 604 is used to obtain the emotional speech loss function using the emotional speech MEL spectrum and the target language speech MEL spectrum with the speaker ID blurred, obtain the speaker speech loss function using the speaker speech MEL spectrum and the MEL spectrum of the target language speech with the emotional ID blurred, obtain the speaker speech loss function, the emotional speech loss function, the generator loss function and the L2 loss function, and obtain the generation loss function according to the speaker speech loss function, the emotional speech loss function, the generator loss function and the L2 loss function, obtain the discriminant loss function, and use the generation loss function and the discriminant loss function to perform adversarial training on the generator and the discriminator of the cross-language emotional speech synthesis model respectively until the model converges to obtain the target cross-language emotional speech synthesis model;

[0119] The synthesis module 605 is used to receive the text to be synthesized, input the text to be synthesized into the target cross-language emotional speech synthesis model, and obtain the target speech.

[0120] In order to solve the above technical problems, an embodiment of the present application further provides a computer device, which adopts the following technical solution: including a processor, a network module and a memory, and the processor and the memory are interconnected through the network module.

[0121] The computer device may be a computer, server, workstation or other device, or a mobile phone, tablet, vehicle-mounted mobile terminal or other device with program execution capability. The internal structure diagram of the computer device may be as follows: Fig.10 As shown, Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. The computer device includes a processor, a memory and a network module. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, instructions or codes. The internal memory provides an environment for the operation of the operating system and instructions or codes in the non-volatile storage medium. When the instructions or codes are executed by the processor, the functions or steps of the above-mentioned cross-language emotional speech synthesis method based on ByT5 are implemented. The network module of the computer device may include a network interface and / or a wireless network module, and the computer device may communicate with other devices or service platforms through the network module. In addition, the computer device may also include a display screen and an input device, etc.

[0122] Among them, the memory is used to store a computer program, the computer program includes program instructions, the processor is configured to call the program instructions, and when the processor executes the instructions or codes, the steps of the cross-language emotional speech synthesis method based on ByT5 as described above are implemented.

[0123] In order to solve the above problems, the embodiment of the present application further proposes a computer-readable storage medium, which adopts the following technical solution:

[0124] The computer readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by the processor, Figures 1 to 7 The cross-language emotional speech synthesis method based on ByT5 provided in each step can be specifically referred to the implementation method provided in the above steps, which will not be repeated here.

[0125] The computer-readable storage medium may be a ByT5-based cross-language emotional speech synthesis device provided in any of the aforementioned embodiments or an internal storage unit of the terminal device, such as a hard disk or memory of a computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device.

[0126] Furthermore, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0127] However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0128] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0129] Compared with the prior art, the embodiments of the present application mainly have the following technical effects: by using BYT5 to convert the bit sequence of the input text and obtain the text feature vector, the text is stored in the form of bytes in the computer device, and any text and vocabulary phrases can be represented by bytes or byte sequences, making full use of the byte encoding of the ByT5 architecture to represent the advantages in solving multilingual and new word problems, solving the problem of the token-based model being restricted by the dictionary, and improving the robustness of the model. By integrating feature information such as part of speech, syntactic structure, speaker, emotion, etc., the quality and naturalness of the synthesized speech are effectively improved, the speaker characteristics and emotional characteristics of the speech are controllable, and the emotional and speaker characteristics obtained after decoding are used to enable the model to learn these features more effectively. By conducting adversarial training on the model, the interference of information noise in the modeling process is reduced, and the quality of the synthesized speech is improved.

[0130] Compared with the prior art, the embodiments of the present application mainly have the following technical effects: by constructing a parameter efficient fine-tuning super network, different types of parameter fine-tuning modules can be used at the same time, so that the pre-trained model can adaptively select the best parameter efficient fine-tuning module according to the task, thereby unlocking the capabilities of the large model to the greatest extent and improving the average performance of the deployed model on various medical tasks.

[0131] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0132] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cross-language emotional speech synthesis method based on ByT5, characterized by comprising: Obtain the part-of-speech information embedding vector and syntactic information embedding vector of the input text, use BYT5 to convert the bit sequence of the input text, and obtain the text feature vector; Obtaining an emotion ID embedding vector and a speaker ID embedding vector according to the emotion ID information and the speaker ID information respectively, performing feature fusion on the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the emotion ID embedding vector respectively, obtaining an emotion text latent vector, performing feature fusion on the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the speaker ID feature vector respectively, obtaining a speaker text latent vector; Generate an emotional speech MEL spectrum and a speaker speech MEL spectrum according to the emotional text latent vector and the speaker text latent vector respectively, generate a target MEL spectrum using the emotional speech MEL spectrum and the speaker speech MEL spectrum, and use a discriminator to discriminate the target MEL spectrum; The emotional speech MEL spectrum and the target language speech MEL spectrum with the speaker ID blurred are used to obtain the emotional speech loss function, the speaker speech MEL spectrum and the MEL spectrum of the target language speech with the emotional ID blurred are used to obtain the speaker speech loss function, the generation loss function is obtained according to the speaker speech loss function, the emotional speech loss function, the generator loss function and the L2 loss function, the discriminant loss function is obtained, and the generator and the discriminant of the cross-language emotional speech synthesis model are respectively trained adversarially using the generation loss function and the discriminant loss function until the model converges to obtain the target cross-language emotional speech synthesis model; The text to be synthesized is received, and the text to be synthesized is input into the target cross-language emotional speech synthesis model to obtain the target speech.

2. The cross-language emotional speech synthesis method based on ByT5 according to claim 1, characterized in that: The step of obtaining the part-of-speech information embedding vector of the input text includes: Preprocessing the input text, using a part-of-speech tagging tool to perform part-of-speech tagging on each word in the preprocessed text, and obtaining a part-of-speech tagging table; Each word and part-of-speech combination in the part-of-speech tagging table is represented by a vector according to a specified vector dimension to obtain the part-of-speech information embedding vector.

3. The cross-language emotional speech synthesis method based on ByT5 according to claim 1, characterized in that: The step of obtaining a syntactic information embedding vector of the input text includes: Preprocessing the input text, performing syntactic analysis on the preprocessed text using a syntactic analysis tool, and obtaining a syntactic tree; Performing feature extraction on the syntax tree to obtain syntax features; Each syntactic feature is represented by a vector according to a specified vector dimension to obtain the syntactic information embedding vector.

4. The cross-language emotional speech synthesis method based on ByT5 according to claim 1, characterized in that: The step of respectively fusing the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the emotion ID embedding vector to obtain the emotion text latent vector comprises: Preprocessing the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the sentiment ID embedding vector to keep the dimensions of all embedding vectors consistent; The preprocessed part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the emotion ID embedding vector are concatenated to obtain a first emotion speech long vector; Inputting the first emotional speech long vector into a fully connected layer for linear operation to obtain a second emotional speech long vector; The second emotional speech long vector is input into the latent function layer, and nonlinear transformation is performed to obtain the emotional text latent vector.

5. The cross-language emotional speech synthesis method based on ByT5 according to claim 1, characterized in that: The step of respectively fusing the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the speaker ID feature vector to obtain the speaker text latent vector includes: Preprocessing the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the speaker ID embedding vector to make the dimensions of all embedding vectors consistent; Concatenate the preprocessed part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the speaker ID embedding vector to obtain a first speaker speech long vector; Inputting the first speaker's speech long vector into a fully connected layer for linear operation to obtain a second speaker's speech long vector; The second speaker's speech long vector is input into the latent function layer and nonlinearly transformed to obtain the speaker's text latent vector.

6. The cross-language emotional speech synthesis method based on ByT5 according to claim 1, characterized in that: The method of obtaining the emotional speech loss function by using the emotional speech MEL spectrum and the target language speech MEL spectrum with the speaker ID blurred includes: Acquire a first target language speech corresponding to the input text, and use a translation function to translate the speech formant of the first target language speech to obtain a second target language speech, wherein the speaker ID of the second target language speech is blurred; The MEL spectrum and the emotional speech MEL spectrum of the second target language speech are obtained, and the emotional speech loss function is obtained according to the MEL spectrum and the emotional speech MEL spectrum of the second target language speech.

7. The cross-language emotional speech synthesis method based on ByT5 according to claim 1, characterized in that: The method of obtaining the speaker speech loss function by using the speaker speech MEL spectrum and the MEL spectrum of the target language speech whose emotion ID is blurred includes: Acquire a first target language speech corresponding to the input text, and use an elimination function to eliminate the emotion information of the first target language speech to obtain a third target language speech, wherein the emotion ID of the third target language speech is blurred; The MEL spectrum of the third target language speech and the MEL spectrum of the speaker speech are obtained, and the speaker speech loss function is obtained according to the MEL spectrum of the third target language speech and the MEL spectrum of the speaker speech.

8. A cross-language emotional speech synthesis device based on ByT5, using the cross-language emotional speech synthesis method based on ByT5 according to any one of claims 1 to 7, characterized in that: include: The first acquisition module is used to obtain the part-of-speech information embedding vector and the syntactic information embedding vector of the input text, and convert the bit sequence of the input text using BYT5 to obtain the text feature vector; The second acquisition module is used to obtain the emotion ID embedding vector and the speaker ID embedding vector according to the emotion ID information and the speaker ID information, respectively, and to fuse the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the emotion ID embedding vector to obtain the emotion text latent vector, and to fuse the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the speaker ID feature vector to obtain the speaker text latent vector; A generation module, used to generate an emotional speech MEL spectrum and a speaker speech MEL spectrum according to an emotional text latent vector and a speaker text latent vector, generate a target MEL spectrum using the emotional speech MEL spectrum and the speaker speech MEL spectrum, and discriminate the target MEL spectrum using a discriminator; A training module, for obtaining an emotional speech loss function using the emotional speech MEL spectrum and the MEL spectrum of the target language speech with the speaker ID blurred, obtaining a speaker speech loss function using the speaker speech MEL spectrum and the MEL spectrum of the target language speech with the emotional ID blurred, obtaining a speaker speech loss function, an emotional speech loss function, a generator loss function and an L2 loss function, and obtaining a generation loss function based on the speaker speech loss function, the emotional speech loss function, the generator loss function and the L2 loss function, obtaining a discriminant loss function, and using the generation loss function and the discriminant loss function to perform adversarial training on the generator and the discriminator of the cross-language emotional speech synthesis model respectively until the model converges to obtain a target cross-language emotional speech synthesis model; The synthesis module is used to receive the text to be synthesized, input the text to be synthesized into the target cross-language emotional speech synthesis model, and obtain the target speech.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the cross-language emotional speech synthesis method based on ByT5 as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the cross-language emotional speech synthesis method based on ByT5 as described in any one of claims 1 to 7 are implemented.

Images