Byt5-based cross-lingual affective speech synthesis method and device, equipment and medium

Abstract

The application belongs to the technical field of intelligent speech synthesis, and discloses a cross-language emotional speech synthesis method, device, equipment and medium based on ByT5, which comprises the following steps: obtaining a part-of-speech information embedding vector, a syntax information embedding vector and a text feature vector of an input text; performing feature fusion to obtain an emotional text hidden vector and a speaker text hidden vector; generating an emotional speech MEL spectrum and a speaker speech MEL spectrum according to the emotional text hidden vector and the speaker text hidden vector respectively, generating a target MEL spectrum by using the emotional speech MEL spectrum and the speaker speech MEL spectrum, and discriminating the target MEL spectrum by using a discriminator; and performing adversarial training on a generator and the discriminator of a cross-language emotional speech synthesis model by using a generation loss function and a discrimination loss function until the model converges. The application solves the problem that a token-based model is limited by a dictionary, improves the robustness of the model, effectively improves the quality and naturalness of synthesized speech, and makes the speaker features and emotional features of the speech controllable.

Description

Technical Field

[0001] This invention relates to the field of intelligent speech synthesis technology, and in particular to a cross-language emotional speech synthesis method, apparatus, device, and medium based on ByT5. Background Technology

[0002] Cross-lingual speech synthesis involves converting text in one language into natural, fluent speech in another. This task holds significant promise for applications in multilingual societies and international contexts. Expanding cross-lingual capabilities with speech synthesis features that incorporate emotion and speaker-specific characteristics can further improve the user experience of voice interaction systems while providing a wider range of services.

[0003] Currently, most TTS systems rely on tokens for text processing. However, this approach has several drawbacks: the model can only recognize and process characters within the vocabulary (i.e., those with corresponding tokens). For unknown characters, the model often cannot process them. Furthermore, existing cross-language speech synthesis methods often produce poor speech quality and suffer from semantic loss. Summary of the Invention

[0004] The purpose of this application is to propose a cross-language emotional speech synthesis method, apparatus, device and medium based on ByT5 to solve the problems of limited recognition range and poor synthesized speech effect of existing speech models.

[0005] To address the aforementioned technical problems, this application provides a cross-lingual emotional speech synthesis method based on ByT5, employing the following technical solution:

[0006] Obtain the part-of-speech information embedding vector and the syntactic information embedding vector of the input text, and use BYT5 to transform the bit sequence of the input text to obtain the text feature vector;

[0007] Based on the sentiment ID information and the speaker ID information, obtain the sentiment ID embedding vector and the speaker ID embedding vector respectively. Then, perform feature fusion with the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the sentiment ID embedding vector respectively to obtain the sentiment text latent vector. Finally, perform feature fusion with the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the speaker ID feature vector respectively to obtain the speaker text latent vector.

[0008] Based on the latent vectors of the emotional text and the latent vectors of the speaker text, an emotional speech MEL spectrum and a speaker speech MEL spectrum are generated respectively. A target MEL spectrum is generated using the emotional speech MEL spectrum and the speaker speech MEL spectrum, and a discriminator is used to discriminate the target MEL spectrum.

[0009] An emotional speech loss function is obtained using the emotional speech MEL spectrum and the target language speech MEL spectrum with the speaker ID blurred. A speaker speech loss function is obtained using the speaker speech MEL spectrum and the target language speech MEL spectrum with the emotional ID blurred. A generation loss function is obtained based on the speaker speech loss function, emotional speech loss function, generator loss function, and L2 loss function. A discriminant loss function is obtained. The generator and discriminant of the cross-language emotional speech synthesis model are then subjected to adversarial training using the generation loss function and the discriminant loss function, respectively, until the model converges, thus obtaining the target cross-language emotional speech synthesis model.

[0010] Receive the text to be synthesized, input the text to be synthesized into the target cross-linguistic emotional speech synthesis model, and obtain the target speech.

[0011] Furthermore, obtaining the part-of-speech information embedding vector of the input text includes:

[0012] The input text is preprocessed, and each word in the preprocessed text is tagged with a part-of-speech tagging tool to obtain a part-of-speech tagging table;

[0013] Each word and part-of-speech tagging combination in the part-of-speech tagging table is represented by a vector according to the specified vector dimension, and the part-of-speech information embedding vector is obtained.

[0014] Furthermore, obtaining the syntactic information embedding vector of the input text includes:

[0015] The input text is preprocessed, and the preprocessed text is then parsed using a syntactic analysis tool to obtain a syntactic tree.

[0016] Feature extraction is performed on the syntax tree to obtain syntactic features;

[0017] Each syntactic feature is represented by a vector according to the specified vector dimension to obtain the syntactic information embedding vector.

[0018] Furthermore, the step of fusing the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the sentiment ID embedding vector to obtain the sentiment text latent vector includes:

[0019] The part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the sentiment ID embedding vector are preprocessed to ensure that the dimensions of all embedding vectors are consistent.

[0020] The preprocessed part-of-speech information embedding vector, syntactic information embedding vector, text feature vector, and sentiment ID embedding vector are concatenated to obtain the first sentiment speech long vector.

[0021] The first emotional speech long vector is input into a fully connected layer for linear operation to obtain the second emotional speech long vector.

[0022] The second emotional speech long vector is input into the hidden function layer and subjected to nonlinear transformation to obtain the emotional text hidden vector.

[0023] Furthermore, the step of fusing the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the speaker ID feature vector to obtain the speaker's text latent vector includes:

[0024] The part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the speaker ID embedding vector are preprocessed to ensure that the dimensions of all embedding vectors are consistent.

[0025] The preprocessed part-of-speech information embedding vector, syntactic information embedding vector, text feature vector, and speaker ID embedding vector are concatenated to obtain the first speaker's long speech vector.

[0026] The first speaker's long speech vector is input into a fully connected layer for linear operation to obtain the second speaker's long speech vector.

[0027] The second speaker's long speech vector is input into the implicit function layer and subjected to nonlinear transformation to obtain the speaker's text latent vector.

[0028] Furthermore, the step of obtaining the emotional speech loss function using the emotional speech MEL spectrum and the target language speech MEL spectrum with the speaker ID blurred includes:

[0029] The first target language speech corresponding to the input text is obtained. The speech formants of the first target language speech are shifted using a translation function to obtain the second target language speech. The speaker ID of the second target language speech is blurred.

[0030] Obtain the MEL spectrum and emotional speech MEL spectrum of the second target language speech, and obtain the emotional speech loss function based on the MEL spectrum and emotional speech MEL spectrum of the second target language speech.

[0031] Furthermore, the step of obtaining the speaker's speech loss function using the speaker's speech MEL spectrum and the MEL spectrum of the target language speech with the emotion ID blurred includes:

[0032] The first target language speech corresponding to the input text is obtained, and the emotional information of the first target language speech is eliminated using an elimination function to obtain the third target language speech. The emotional ID of the third target language speech is blurred.

[0033] Obtain the MEL spectrum of the third target language speech and the speaker's speech MEL spectrum, and obtain the speaker's speech loss function based on the MEL spectrum of the third target language speech and the speaker's speech MEL spectrum.

[0034] To address the aforementioned issues, a cross-lingual emotional speech synthesis device based on ByT5 is also provided, employing the aforementioned cross-lingual emotional speech synthesis method based on ByT5, including:

[0035] The first acquisition module is used to acquire the part-of-speech information embedding vector and the syntactic information embedding vector of the input text, and to convert the bit sequence of the input text using BYT5 to obtain the text feature vector;

[0036] The second acquisition module is used to acquire the sentiment ID embedding vector and the speaker ID embedding vector respectively based on the sentiment ID information and the speaker ID information, and to perform feature fusion of the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the sentiment ID embedding vector respectively to acquire the sentiment text latent vector, and to perform feature fusion of the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the speaker ID feature vector respectively to acquire the speaker text latent vector;

[0037] The generation module is used to generate emotional speech MEL spectra and speaker speech MEL spectra based on the emotional text latent vector and speaker text latent vector, respectively; generate a target MEL spectra using the emotional speech MEL spectra and speaker speech MEL spectra; and use a discriminator to discriminate the target MEL spectra.

[0038] The training module is used to obtain an emotional speech loss function using the emotional speech MEL spectrum and the target language speech MEL spectrum with the speaker ID blurred; to obtain a speaker speech loss function using the speaker speech MEL spectrum and the target language speech MEL spectrum with the emotional ID blurred; to obtain the speaker speech loss function, emotional speech loss function, generator loss function, and L2 loss function; to obtain a generation loss function based on the speaker speech loss function, emotional speech loss function, generator loss function, and L2 loss function; to obtain a discriminant loss function; and to perform adversarial training on the generator and discriminator of the cross-language emotional speech synthesis model using the generation loss function and the discriminant loss function, respectively, until the model converges, thus obtaining the target cross-language emotional speech synthesis model.

[0039] The synthesis module is used to receive the text to be synthesized, input the text to be synthesized into the target cross-linguistic emotional speech synthesis model, and obtain the target speech.

[0040] To address the aforementioned technical problems, this application also provides a computer device, including a memory and a processor. The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the ByT5-based cross-language emotional speech synthesis method described above.

[0041] To address the aforementioned technical problems, this application also provides a computer-readable storage medium storing computer-readable instructions. When executed by a processor, these computer-readable instructions implement the steps of the ByT5-based cross-language emotional speech synthesis method described above.

[0042] Compared with existing technologies, the embodiments of this application have the following main technical effects: By utilizing BYT5 to transform the bit sequence of the input text and obtaining the text feature vector, text is stored in the form of bytes in computer devices. Any text and vocabulary phrase can be represented by bytes or byte sequences, fully utilizing the advantages of byte encoding in the BYT5 architecture in solving multilingual and unfamiliar word problems. This solves the problem of token-based models being limited by dictionaries and improves the robustness of the model. By fusing feature information such as part-of-speech, syntactic structure, speaker, and emotion, the quality and naturalness of the synthesized speech are effectively improved. The speaker features and emotional features of the speech are controllable, and the emotional and speaker features obtained after decoding enable the model to learn these features more effectively. By conducting adversarial training on the model, the interference of information noise during the modeling process is reduced, and the quality of the synthesized speech is improved. Attached Figure Description

[0043] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 A flowchart of an embodiment of a cross-language emotional speech synthesis method based on ByT5 according to this application;

[0045] Figure 2 yes Figure 1 A flowchart of a specific implementation of step 100, obtaining the part-of-speech information embedding vector;

[0046] Figure 3 yes Figure 1 A flowchart of a specific implementation of step 100, obtaining the syntactic information embedding vector;

[0047] Figure 4 yes Figure 1 A flowchart of a specific implementation method for obtaining the latent vector of sentiment text in step 200;

[0048] Figure 5 yes Figure 1 A flowchart of a specific implementation of the speaker's text latent vector in step 200;

[0049] Figure 6 yes Figure 1 A flowchart of a specific implementation of step 400, obtaining the emotional speech loss function;

[0050] Figure 7 yes Figure 1 A flowchart of a specific implementation of step 400, which involves obtaining the speaker's speech loss function;

[0051] Figure 8 This is a flowchart illustrating a cross-lingual emotional speech synthesis method based on ByT5 according to this application;

[0052] Figure 9 This is a schematic diagram of the structure of an embodiment of a cross-language emotional speech synthesis device based on ByT5 according to this application;

[0053] Figure 10 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0054] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are all within the scope of protection of this invention.

[0055] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings of this application are intended to cover non-exclusive inclusion.

[0057] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0058] The cross-language emotion-based speech synthesis method based on ByT5 provided in this application can be applied to computer devices with instruction or program execution capabilities. These computer devices can be, but are not limited to, various servers, workstations, personal computers, laptops, etc. The difference in execution between different computing devices is merely a difference in the execution entity of the solution; those skilled in the art will foresee that running the solution on different computing devices will produce the same technical effect. The invention will now be described in detail through specific embodiments.

[0059] The purpose of this application is to propose a cross-language emotional speech synthesis method, apparatus, device and medium based on ByT5 to solve the problems of limited recognition range and poor synthesized speech effect of existing speech models.

[0060] Text-to-Speech (TTS) models are technologies that convert input text information into natural and fluent speech output. Common TTS models include VITS (Variational Inference with Adversarial Learning for End-to-End Text-to-Speech), YourTTS, PhomeTTS, and StyleTTS2. These models all rely on tokens for text processing. A key technical challenge is that these models can only recognize and process characters within their vocabulary (i.e., those with corresponding tokens). They often cannot process unknown words, resulting in less than satisfactory speech synthesis performance.

[0061] To address the aforementioned technical problems, this application provides a cross-lingual emotional speech synthesis method based on ByT5, employing the following technical solution: Figure 1 , Figure 1 A flowchart of an embodiment of a cross-lingual emotion-based speech synthesis method based on ByT5 according to this application includes:

[0062] S100. Obtain the part-of-speech information embedding vector and the syntactic information embedding vector of the input text. Use BYT5 to transform the bit sequence of the input text to obtain the text feature vector.

[0063] ByT5 (Byte-level T5) is a language model based on the T5 (Text-to-Text Transfer Transformer) architecture. It inherits the basic architecture of T5 and adopts the encoder-decoder structure of the Transformer. This structure performs exceptionally well in natural language processing tasks, effectively capturing the contextual information and semantic relationships of text. Through pre-training at the byte level, the model learns the byte-level representation of text, thus better understanding and generating text. The pre-training process utilizes a large amount of text data, enabling the model to learn common features and patterns across different languages.

[0064] In this embodiment, when obtaining the text feature vector, the input text is first converted into a bit sequence.

[0065] Specifically, input text consists of characters, and each character has a corresponding encoded representation in a computer. Common encoding methods include ASCII and Unicode. These encodings map characters to specific numeric values, which can then be further converted into binary (bit sequence) representations.

[0066] You can choose ASCII encoding, which converts each character into its corresponding numeric code value according to the selected encoding method. For example, in ASCII, the character "A" corresponds to the number 65. Next, the numeric code value of each character is converted into its binary representation. For example, the binary representation of the number 65 is 1000001. Then, concatenating the binary representations of each character sequentially yields the bit sequence of the input text.

[0067] For example, for the text "Hello", assuming ASCII encoding is used: "H" corresponds to the number 72, which is 1001000 in binary. "e" corresponds to the number 101, which is 1100101 in binary. "l" corresponds to the number 108, which is 1101100 in binary. "l" again corresponds to the number 108, which is 1101100 in binary. "o" corresponds to the number 111, which is 1101111 in binary. Concatenating these binary sequences, we get the bit sequence of "Hello" as 10010001100101110110011011001101111.

[0068] In this embodiment, the input text is converted into byte form and stored in a computer device. Any text and word phrase can be represented by bytes or byte sequences. Therefore, the ByT5 architecture, which uses byte encoding to represent words, demonstrates outstanding superiority in solving multilingual and unfamiliar word problems. The cross-lingual emotional speech synthesis using the ByT5 architecture in this application embodiment is more robust and performs better on word-level tasks that are sensitive to spelling and pronunciation.

[0069] In some preferred embodiments, such as Figure 2 , Figure 2 yes Figure 1 A flowchart of a specific implementation of obtaining part-of-speech information embedding vector in step S100; S100 includes: S110, preprocessing the input text, using a part-of-speech tagging tool to tag each word in the preprocessed text with part-of-speech tags, and obtaining a part-of-speech tagging table; S120, performing vector representation on each word and part-of-speech combination in the part-of-speech tagging table according to the specified vector dimension, and obtaining part-of-speech information embedding vector.

[0070] In this embodiment, the input text is first preprocessed, including operations such as removing special characters, standardizing punctuation marks, and unifying capitalization.

[0071] Use a part-of-speech tagging tool to tag each word in the text with its part of speech. The tagging results may include nouns, verbs, adjectives, adverbs, etc.

[0072] In this embodiment, a part-of-speech tagging table is constructed using all the different words and part-of-speech combinations that appear in the input text. In addition, a unique index can be assigned to each word and part-of-speech combination.

[0073] like Figure 8 , Figure 8 This is a flowchart illustrating a cross-lingual emotional speech synthesis method based on ByT5 according to this application;

[0074] like Figure 8The input text is processed by part-of-speech tagging and syntactic analysis tools to obtain part-of-speech information embedding vectors and syntactic information embedding vectors. The bit sequence of the input text is then transformed using BYT5 to obtain the text feature vector.

[0075] Based on the sentiment ID information and speaker ID information, obtain the sentiment ID embedding vector and speaker ID embedding vector respectively. Then, perform feature fusion with the part-of-speech information embedding vector, syntactic information embedding vector, text feature vector and sentiment ID embedding vector respectively to obtain the sentiment text latent vector. Finally, perform feature fusion with the part-of-speech information embedding vector, syntactic information embedding vector, text feature vector and speaker ID feature vector respectively to obtain the speaker text latent vector.

[0076] Based on the latent vectors of the emotional text and the latent vectors of the speaker text, the MEL spectrum of the emotional speech and the MEL spectrum of the speaker speech are generated respectively. The target MEL spectrum is generated using the MEL spectrum of the emotional speech and the MEL spectrum of the speaker speech, and the discriminator is used to discriminate the target MEL spectrum.

[0077] The emotional speech loss function is obtained by using the MEL spectrum of the emotional speech and the MEL spectrum of the target language speech with the speaker ID blurred. The speaker speech loss function is obtained by using the MEL spectrum of the speaker speech and the MEL spectrum of the target language speech with the emotion ID blurred. The generation loss function is obtained based on the speaker speech loss function, the emotional speech loss function, the generator loss function, and the L2 loss function. The generator and discriminator of the cross-language emotional speech synthesis model are subjected to adversarial training using the generation loss function and the discriminant loss function, respectively, until the model converges, thus obtaining the target cross-language emotional speech synthesis model.

[0078] In this embodiment, by obtaining part-of-speech tag (POT) embedding vectors and combining them with other features, richer input features can be provided to the natural language processing model, thereby improving the model's performance. For example, in named entity recognition tasks, POT information can help the model better identify entity types. It can also help the model better generalize to different text domains and language variations. Since POTs have a certain degree of stability across different texts, introducing POT information can make the model more robust when facing new input texts.

[0079] In some alternative implementations, such as Figure 3 , Figure 3 yes Figure 1The flowchart shows a specific implementation of obtaining the syntactic information embedding vector in step S100. Step S100 includes: S130, preprocessing the input text, performing syntactic analysis on the preprocessed text using a syntactic analysis tool, and obtaining a syntactic tree; S140, extracting features from the syntactic tree to obtain syntactic features; and S150, representing each syntactic feature as a vector according to a specified vector dimension to obtain the syntactic information embedding vector.

[0080] In this embodiment, the input text is first preliminarily cleaned to remove unnecessary noise, such as special characters and garbled text. Then, a syntactic analysis tool is used to perform syntactic analysis on the text. The analysis results will provide the sentence's component structure, such as subject-verb-object, attributive, adverbial, and complement relationships, as well as the dependency relationships between words.

[0081] In this embodiment, a syntactic tree representation of the sentence is constructed based on the results of syntactic analysis. The syntactic tree clearly displays the structural hierarchy of the sentence and the relationships between words. Useful features, such as node type, node depth, and subtree structure, are extracted from the syntactic tree.

[0082] During training, the weights of the vectors are continuously adjusted so that sentences with similar syntactic structures are closer together in the vector space.

[0083] In this embodiment, by obtaining syntactic information embedding vectors, additional and valuable features are provided for ByT5-based cross-lingual emotional speech synthesis. Combined with part-of-speech embedding vector features, the expressive power of the ByT5-based cross-lingual emotional speech synthesis model can be significantly enhanced. For example, in text classification tasks, syntactic information embedding vectors can help the model better distinguish different types of text structures, thereby improving classification accuracy. Furthermore, it can achieve better performance on different datasets and tasks, reducing over-reliance on domain-specific data.

[0084] S200. Obtain the sentiment ID embedding vector and speaker ID embedding vector respectively based on the sentiment ID information and speaker ID information. Perform feature fusion with the part-of-speech information embedding vector, syntactic information embedding vector, text feature vector and sentiment ID embedding vector respectively to obtain the sentiment text latent vector. Perform feature fusion with the part-of-speech information embedding vector, syntactic information embedding vector, text feature vector and speaker ID feature vector respectively to obtain the speaker text latent vector.

[0085] In some alternative implementations, such as Figure 4 , Figure 4 yes Figure 1 A flowchart of a specific implementation method for obtaining the latent vector of sentiment text in S200, wherein S200 includes:

[0086] S210. Preprocess the part-of-speech embedding vector, syntactic information embedding vector, text feature vector, and sentiment ID embedding vector to ensure that the dimensions of all embedding vectors are consistent. S220. Concatenate the preprocessed part-of-speech embedding vector, syntactic information embedding vector, text feature vector, and sentiment ID embedding vector to obtain the first sentiment speech long vector. S230. Input the first sentiment speech long vector into a fully connected layer for linear operation to obtain the second sentiment speech long vector. S24. Input the second sentiment speech long vector into a hidden function layer for nonlinear transformation to obtain the sentiment text latent vector.

[0087] In this embodiment, the acquired part-of-speech embedding vectors, syntactic information embedding vectors, text feature vectors, and sentiment ID embedding vectors are preprocessed to ensure that the dimensions and data types of each feature are consistent. Specific preprocessing includes removing outliers, handling missing values, and normalizing the vectors. After preprocessing, these feature vectors are concatenated to form a long vector, which is then used as the input to the fully connected layer. The weight matrix and bias terms of the fully connected layer are initialized using common initialization methods such as Xavier initialization or He initialization. The concatenated first sentiment speech long vector is then input into the fully connected layer. The fully connected layer performs linear operations, multiplying the feature vector by the weight matrix and adding the bias term to obtain a new vector, which is the second sentiment speech long vector.

[0088] In this embodiment, through the above steps, the fully connected layer can effectively fuse part-of-speech information embedding vectors, syntactic information embedding vectors, text feature vectors, and sentiment ID embedding vectors to obtain more powerful latent vectors, providing better feature representations for subsequent tasks.

[0089] In some alternative implementations, such as Figure 5 , Figure 5 yes Figure 1 A flowchart of a specific implementation of the S200 speaker text latent vector, including:

[0090] S250. Preprocess the part-of-speech embedding vector, syntactic information embedding vector, text feature vector, and speaker ID embedding vector to ensure that the dimensions of all embedding vectors are consistent. S260. Concatenate the preprocessed part-of-speech embedding vector, syntactic information embedding vector, text feature vector, and speaker ID embedding vector to obtain the first speaker's long speech vector. S270. Input the first speaker's long speech vector into a fully connected layer for linear operation to obtain the second speaker's long speech vector. S28. Input the second speaker's long speech vector into a hidden function layer for nonlinear transformation to obtain the speaker's text latent vector.

[0091] In this embodiment, the acquired part-of-speech embedding vectors, syntactic embedding vectors, text feature vectors, and speaker ID embedding vectors are preprocessed to ensure that the dimensions and data types of each feature are consistent. Specific preprocessing includes removing outliers, handling missing values, and normalizing the vectors. After preprocessing, these feature vectors are concatenated to form a long vector, which is then used as the input to the fully connected layer. The weight matrix and bias terms of the fully connected layer are initialized using common initialization methods such as Xavier initialization or He initialization. The concatenated long vector of the first speaker's speech is then input into the fully connected layer. The fully connected layer performs linear operations, multiplying the feature vector by the weight matrix and adding the bias term to obtain a new vector, which is the long vector of the second speaker's speech.

[0092] In this embodiment, through the above steps, the fully connected layer can effectively fuse the part-of-speech information embedding vector, syntactic information embedding vector, text feature vector, and speaker ID embedding vector to obtain a more powerful latent vector, providing a better feature representation for subsequent tasks.

[0093] S300. Generate the emotional speech MEL spectrum and the speaker speech MEL spectrum based on the emotional text latent vector and the speaker text latent vector, respectively. Generate the target MEL spectrum using the emotional speech MEL spectrum and the speaker speech MEL spectrum, and use a discriminator to discriminate the target MEL spectrum.

[0094] like Figure 8 The latent vectors of the emotional text and the latent vectors of the speaker text are input into the corresponding generator of the cross-language emotional speech synthesis model in this application embodiment to generate the emotional speech MEL spectrum and the speaker speech MEL spectrum, respectively. Then, the generated emotional speech MEL spectrum and speaker speech MEL spectrum are processed by the decoder of the model to generate the target MEL spectrum of the target language. The generated target MEL spectrum is then compared with the real MEL spectrum for discrimination and classification.

[0095] S400. Obtain the emotional speech loss function using the emotional speech MEL spectrum and the target language speech MEL spectrum with the speaker ID blurred. Obtain the speaker speech loss function using the speaker speech MEL spectrum and the target language speech MEL spectrum with the emotional ID blurred. Obtain the generation loss function based on the speaker speech loss function, emotional speech loss function, generator loss function and L2 loss function. Obtain the discriminant loss function. Use the generation loss function and discriminant loss function to perform adversarial training on the generator and discriminator of the cross-language emotional speech synthesis model until the model converges, and obtain the target cross-language emotional speech synthesis model.

[0096] In some alternative implementations, such as Figure 6 , Figure 6 yes Figure 1 A flowchart illustrating a specific implementation of the S400 method for obtaining the emotional speech loss function; S400 includes:

[0097] S410. Obtain the first target language speech corresponding to the input text, and use a translation function to shift the speech formants of the first target language speech to obtain the second target language speech. The speaker ID of the second target language speech is blurred. S420. Obtain the MEL spectrum and emotional speech MEL spectrum of the second target language speech, and obtain the emotional speech loss function based on the MEL spectrum and emotional speech MEL spectrum of the second target language speech.

[0098] In this embodiment, an "information fuzzing" approach is used to better separate speaker information and emotional information in speech.

[0099] Since the speaker's timbre is closely related to the formants of the speech, in this embodiment, the speaker's identity information in the speech signal is disturbed by interfering with the formants of the speech, while the emotional expression is expected to remain unchanged.

[0100] In this embodiment, a translation function f is used. p The formants are shifted within a random range, thereby eliminating speaker information from the original speech. Ultimately, the target language speech with blurred speaker information and its corresponding Mel spectrum are obtained, namely the Mel spectrum of the second target language speech and the Mel spectrum of the emotional speech.

[0101] In this implementation, by adopting an "information fuzzy" approach to obtain emotional information and an emotional speech loss function, the process of generating a specified emotion in the cross-language emotional speech synthesis model based on ByT5 becomes more controllable.

[0102] In some alternative implementations, such as Figure 7 , Figure 7 yes Figure 1 A flowchart of a specific implementation of the speaker speech loss function obtained in step S400; S400 includes: S430, obtaining the first target language speech corresponding to the input text, using an elimination function to eliminate the emotional information of the first target language speech, obtaining the third target language speech, the emotional ID of the third target language speech being blurred; S440, obtaining the MEL spectrum of the third target language speech and the speaker speech MEL spectrum, and obtaining the speaker speech loss function based on the MEL spectrum of the third target language speech and the speaker speech MEL spectrum.

[0103] In this embodiment, an "information fuzzing" approach is used to better separate speaker information and emotional information in speech.

[0104] In this implementation, based on the close relationship between emotional expression and pitch and pitch trend, pitch information is perturbed by random shifting and averaging.

[0105] Specifically, in this embodiment, the pitch of the speech is first shifted by a random amplitude. Since the pitch variation trend in different emotional categories of speech is still preserved after the shift, all pitch values ​​are further monotonicized to the average pitch. The process of eliminating emotional information is performed using the function f. h This indicates that the target language speech with blurred emotional information and the corresponding Mel spectrum were obtained, namely the Mel spectrum of the third target language speech and the speaker's Mel spectrum.

[0106] In this implementation, by adopting an "information fuzziness" approach to obtain speaker information and the corresponding speaker speech loss function, the process of generating specified emotions by the ByT5-based cross-lingual emotion speech synthesis model becomes more controllable.

[0107] S500: Receive the text to be synthesized, input the text to be synthesized into the target cross-linguistic emotional speech synthesis model, and obtain the target speech.

[0108] In this embodiment of the application, when training the cross-language emotional speech synthesis model based on ByT5, an adversarial training strategy is adopted, in which the generated target speech Mel spectrum and the real Mel spectrum are input into the discriminator to distinguish between real and fake.

[0109] Assume the emotional speech loss function and the speaker speech loss function are L... emotion L speaker The loss function of the Mel spectrum generator is L Mel The L2 loss function between the generated target Mel spectrum and the true Mel spectrum is L rec Then the loss function L for training the discriminator D for::

[0110]

[0111] Loss function L for training the generator G for:

[0112]

[0113] in, α, β, γ, and δ are hyperparameters for adjusting the weights of each loss function, and can be adjusted according to the actual training situation.

[0114] To address the aforementioned technical problems, this application also provides a cross-language emotion-based speech synthesis device 600 based on ByT5, such as... Figure 9 , Figure 9This is a schematic diagram of an embodiment of a ByT5-based cross-lingual emotional speech synthesis device 600 according to this application; employing the ByT5-based cross-lingual emotional speech synthesis method of the first aspect, including:

[0115] The first acquisition module 601 is used to acquire the part-of-speech information embedding vector and the syntactic information embedding vector of the input text, and to convert the bit sequence of the input text using BYT5 to obtain the text feature vector.

[0116] The second acquisition module 602 is used to acquire the sentiment ID embedding vector and the speaker ID embedding vector respectively based on the sentiment ID information and the speaker ID information, and to perform feature fusion with the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the sentiment ID embedding vector respectively to acquire the sentiment text latent vector, and to perform feature fusion with the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the speaker ID feature vector respectively to acquire the speaker text latent vector;

[0117] The generation module 603 is used to generate the emotional speech MEL spectrum and the speaker speech MEL spectrum based on the emotional text latent vector and the speaker text latent vector, respectively; generate the target MEL spectrum using the emotional speech MEL spectrum and the speaker speech MEL spectrum; and use a discriminator to discriminate the target MEL spectrum.

[0118] Training module 604 is used to obtain the emotional speech loss function using the MEL spectrum of the emotional speech and the MEL spectrum of the target language speech with the speaker ID blurred, obtain the speaker speech loss function using the MEL spectrum of the speaker speech and the MEL spectrum of the target language speech with the emotional ID blurred, obtain the speaker speech loss function, the emotional speech loss function, the generator loss function and the L2 loss function, obtain the generation loss function based on the speaker speech loss function, the emotional speech loss function, the generator loss function and the L2 loss function, obtain the discriminant loss function, and use the generation loss function and the discriminant loss function to perform adversarial training on the generator and discriminator of the cross-language emotional speech synthesis model respectively until the model converges, and obtain the target cross-language emotional speech synthesis model;

[0119] The synthesis module 605 is used to receive the text to be synthesized, input the text to be synthesized into the target cross-linguistic emotional speech synthesis model, and obtain the target speech.

[0120] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution: it includes a processor, a network module, and a memory, with the processor and memory interconnected via the network module.

[0121] This computer device can be a computer, server, workstation, or other similar device; it can also be a mobile device such as a mobile phone, tablet, or in-vehicle mobile terminal; or other devices with program execution capabilities. Its internal structure diagram can be as follows: Figure 10 As shown, Figure 10 This is a schematic diagram of a computer device according to an embodiment of the present application. The computer device includes a processor, a memory, and a network module. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores an operating system, instructions, or code. The internal memory provides an environment for the operation of the operating system and instructions or code in the non-volatile storage media. When the instructions or code are executed by the processor, they implement the functions or steps of the aforementioned ByT5-based cross-language emotional speech synthesis method. The network module of the computer device may include a network interface and / or a wireless network module, through which the computer device can communicate with other devices or service platforms. Furthermore, the computer device may also include a display screen and input devices, etc.

[0122] The memory is used to store computer programs, which include program instructions. The processor is configured to call the program instructions. When the processor executes the instructions or code, it implements the steps of the ByT5-based cross-language emotion-based speech synthesis method as described above.

[0123] To address the aforementioned problems, this application also proposes a computer-readable storage medium, employing the following technical solution:

[0124] The computer-readable storage medium stores a computer program, which includes program instructions that are implemented when executed by a processor. Figures 1 to 7 The ByT5-based cross-language emotional speech synthesis methods provided in each step are detailed in the implementation methods provided in the above steps, and will not be repeated here.

[0125] The aforementioned computer-readable storage medium can be an internal storage unit of the ByT5-based cross-language emotion-based speech synthesis device provided in any of the foregoing embodiments, or the aforementioned terminal device, such as a hard drive or memory of a computer device. The computer-readable storage medium can also be an external storage device of the computer device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device.

[0126] Furthermore, the computer-readable storage medium may include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0127] However, it should be understood that it is not required to implement all the components shown; more or fewer components may be implemented instead. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0128] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0129] Compared with existing technologies, the embodiments of this application have the following main technical effects: By utilizing BYT5 to transform the bit sequence of the input text and obtaining the text feature vector, text is stored in the form of bytes in computer devices. Any text and vocabulary phrase can be represented by bytes or byte sequences, fully utilizing the advantages of byte encoding in the BYT5 architecture in solving multilingual and unfamiliar word problems. This solves the problem of token-based models being limited by dictionaries and improves the robustness of the model. By fusing feature information such as part-of-speech, syntactic structure, speaker, and emotion, the quality and naturalness of the synthesized speech are effectively improved. The speaker features and emotional features of the speech are controllable, and the emotional and speaker features obtained after decoding enable the model to learn these features more effectively. By conducting adversarial training on the model, the interference of information noise during the modeling process is reduced, and the quality of the synthesized speech is improved.

[0130] Compared with the prior art, the embodiments of this application have the following main technical effects: by constructing a parameter-efficient fine-tuning supernetwork, different types of parameter fine-tuning modules can be used simultaneously, thereby enabling the pre-trained model to adaptively select the best parameter-efficient fine-tuning module according to the task, maximizing the ability of large models and improving the average performance of the deployed model on various medical tasks.

[0131] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0132] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cross-lingual emotion-based speech synthesis method based on ByT5, characterized in that it includes: Obtain the part-of-speech information embedding vector and the syntactic information embedding vector of the input text, and use BYT5 to transform the bit sequence of the input text to obtain the text feature vector; Based on the sentiment ID information and the speaker ID information, obtain the sentiment ID embedding vector and the speaker ID embedding vector respectively. Then, perform feature fusion with the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the sentiment ID embedding vector respectively to obtain the sentiment text latent vector. Finally, perform feature fusion with the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the speaker ID embedding vector respectively to obtain the speaker text latent vector. Based on the latent vectors of the emotional text and the latent vectors of the speaker text, the MEL spectrum of the emotional speech and the MEL spectrum of the speaker speech are generated respectively. The target MEL spectrum is generated using the MEL spectrum of the emotional speech and the MEL spectrum of the speaker speech, and the discriminator is used to discriminate the target MEL spectrum. An emotional speech loss function is obtained using the emotional speech MEL spectrum and the target language speech MEL spectrum with the speaker ID blurred. A speaker speech loss function is obtained using the speaker speech MEL spectrum and the target language speech MEL spectrum with the emotional ID blurred. A generation loss function and a discriminant loss function are obtained based on the speaker speech loss function, emotional speech loss function, generator loss function, and L2 loss function. The generator and discriminant of the cross-language emotional speech synthesis model are then subjected to adversarial training using the generation loss function and discriminant loss function, respectively, until the model converges, thus obtaining the target cross-language emotional speech synthesis model. Receive the text to be synthesized, input the text to be synthesized into the target cross-linguistic emotional speech synthesis model, and obtain the target speech.

2. The cross-language emotion-based speech synthesis method based on ByT5 according to claim 1, characterized in that, The step of obtaining the part-of-speech information embedding vector of the input text includes: The input text is preprocessed, and each word in the preprocessed text is tagged with a part-of-speech tagging tool to obtain a part-of-speech tagging table; Each word and part-of-speech tagging combination in the part-of-speech tagging table is represented by a vector according to the specified vector dimension, and the part-of-speech information embedding vector is obtained.

3. The cross-language emotion-based speech synthesis method based on ByT5 according to claim 1, characterized in that, The process of obtaining the syntactic information embedding vector of the input text includes: The input text is preprocessed, and the preprocessed text is then parsed using a syntactic analysis tool to obtain a syntactic tree. Feature extraction is performed on the syntax tree to obtain syntactic features; Each syntactic feature is represented by a vector according to the specified vector dimension to obtain the syntactic information embedding vector.

4. The cross-language emotion-based speech synthesis method based on ByT5 according to claim 1, characterized in that, The step of fusing the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the sentiment ID embedding vector to obtain the sentiment text latent vector includes: The part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the sentiment ID embedding vector are preprocessed to ensure that the dimensions of all embedding vectors are consistent. The preprocessed part-of-speech information embedding vector, syntactic information embedding vector, text feature vector, and sentiment ID embedding vector are concatenated to obtain the first sentiment speech long vector. The first emotional speech long vector is input into a fully connected layer for linear operation to obtain the second emotional speech long vector. The second emotional speech long vector is input into the hidden function layer and subjected to nonlinear transformation to obtain the emotional text hidden vector.

5. The cross-language emotion-based speech synthesis method based on ByT5 according to claim 1, characterized in that, The step of fusing the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the speaker ID embedding vector to obtain the speaker's text latent vector includes: The part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector, and the speaker ID embedding vector are preprocessed to ensure that the dimensions of all embedding vectors are consistent. The preprocessed part-of-speech information embedding vector, syntactic information embedding vector, text feature vector, and speaker ID embedding vector are concatenated to obtain the first speaker's long speech vector. The first speaker's long speech vector is input into a fully connected layer for linear operation to obtain the second speaker's long speech vector. The second speaker's long speech vector is input into the implicit function layer and subjected to nonlinear transformation to obtain the speaker's text latent vector.

6. The cross-lingual emotional speech synthesis method based on ByT5 according to claim 1, characterized in that, The step of obtaining the emotional speech loss function using the emotional speech MEL spectrum and the target language speech MEL spectrum with the speaker ID blurred includes: The first target language speech corresponding to the input text is obtained. The speech formants of the first target language speech are shifted using a translation function to obtain the second target language speech. The speaker ID of the second target language speech is blurred. Obtain the MEL spectrum and emotional speech MEL spectrum of the second target language speech, and obtain the emotional speech loss function based on the MEL spectrum and emotional speech MEL spectrum of the second target language speech.

7. The cross-language emotion-based speech synthesis method based on ByT5 according to claim 1, characterized in that, The step of obtaining the speaker speech loss function using the speaker's speech MEL spectrum and the MEL spectrum of the target language speech with the emotion ID blurred includes: The first target language speech corresponding to the input text is obtained, and the emotional information of the first target language speech is eliminated using an elimination function to obtain the third target language speech. The emotional ID of the third target language speech is blurred. Obtain the MEL spectrum of the third target language speech and the speaker's speech MEL spectrum, and obtain the speaker's speech loss function based on the MEL spectrum of the third target language speech and the speaker's speech MEL spectrum.

8. A cross-lingual emotional speech synthesis device based on ByT5, employing the cross-lingual emotional speech synthesis method based on ByT5 as described in any one of claims 1-7, characterized in that, include: The first acquisition module is used to acquire the part-of-speech information embedding vector and the syntactic information embedding vector of the input text, and to convert the bit sequence of the input text using BYT5 to obtain the text feature vector; The second acquisition module is used to acquire the sentiment ID embedding vector and the speaker ID embedding vector respectively based on the sentiment ID information and the speaker ID information, and to perform feature fusion of the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the sentiment ID embedding vector respectively to acquire the sentiment text latent vector, and to perform feature fusion of the part-of-speech information embedding vector, the syntactic information embedding vector, the text feature vector and the speaker ID embedding vector respectively to acquire the speaker text latent vector; The generation module is used to generate emotional speech MEL spectra and speaker speech MEL spectra based on the emotional text latent vector and speaker text latent vector, respectively; generate target MEL spectra using the emotional speech MEL spectra and speaker speech MEL spectra; and use a discriminator to discriminate the target MEL spectra. The training module is used to obtain the emotional speech loss function using the emotional speech MEL spectrum and the target language speech MEL spectrum with the speaker ID blurred, obtain the speaker speech loss function using the speaker speech MEL spectrum and the target language speech MEL spectrum with the emotional ID blurred, obtain the speaker speech loss function, emotional speech loss function, generator loss function and L2 loss function, and obtain the generation loss function and the discriminant loss function based on the speaker speech loss function, emotional speech loss function, generator loss function and L2 loss function, and use the generation loss function and the discriminant loss function to perform adversarial training on the generator and discriminator of the cross-language emotional speech synthesis model respectively until the model converges, and obtain the target cross-language emotional speech synthesis model; The synthesis module is used to receive the text to be synthesized, input the text to be synthesized into the target cross-linguistic emotional speech synthesis model, and obtain the target speech.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the ByT5-based cross-language emotional speech synthesis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the ByT5-based cross-lingual emotional speech synthesis method as described in any one of claims 1 to 7.

Images