Byt5-based speech synthesis method and device, equipment and storage medium

Abstract

The application belongs to the technical field of speech synthesis, and discloses a speech synthesis method and device based on ByT5, equipment and a storage medium, the method comprises the following steps: obtaining a bit sequence of an input text, converting the bit sequence into a text feature vector by using a ByT5 model; obtaining a part-of-speech embedding vector and a syntactic structure embedding vector; summing the text feature vector, the part-of-speech embedding vector and the syntactic structure embedding vector, and then sequentially inputting the sum into a full connection layer and an activation function of a generative adversarial network speech model for feature fusion to obtain text fusion features; and sequentially obtaining target language mel spectrums and target language waveforms by using a decoder and a vocoder of the generative adversarial network speech model. The token-based model is not limited by a dictionary, the original text can be directly converted to a unified bit representation of multiple languages, the information of syntax and structure is fused on the basis of the powerful general processing capability of the ByT5 model framework, and the quality of synthesized speech can be effectively improved.

Description

Technical Field

[0001] This invention relates to the field of speech synthesis technology, and in particular to a speech synthesis method, apparatus, device and storage medium based on ByT5. Background Technology

[0002] Cross-lingual speech synthesis involves converting spoken text in one language into natural, fluent spoken text in another. This task holds significant promise for multilingual societies and international applications, improving the user experience of voice interaction systems and providing a wider range of services. For example, in remote consultations, doctors need to exchange information about patients' conditions and diagnoses. Speech synthesis technology can convert the text information entered by doctors into speech, facilitating communication. For instance, during remote consultations, experts can use speech synthesis to convey their diagnoses and treatment suggestions to other doctors, improving the efficiency and accuracy of the consultation. In remote monitoring, for patients with chronic diseases or those recovering at home, remote monitoring systems can use speech synthesis to regularly send voice reminders, such as reminding patients to take medication on time, measure blood pressure, and blood sugar, while simultaneously feeding back the patient's monitoring data to the doctor, allowing the doctor to promptly understand changes in the patient's condition. Currently, most text-to-speech (TTS) systems rely on tokens for text processing. This approach has drawbacks: the speech synthesis model can only recognize and process characters within its vocabulary (i.e., characters for which corresponding tokens exist). It often fails to handle unknown words. Furthermore, existing cross-language speech synthesis methods frequently produce poor-quality speech with semantic gaps. Summary of the Invention

[0003] The purpose of this application is to propose a speech synthesis method, apparatus, device, and storage medium based on ByT5, so as to solve the problems of existing token-based speech synthesis schemes, such as the limitation of the dictionary, poor speech effect, and semantic loss in the token-based model.

[0004] To address the aforementioned technical problems, this application provides a ByT5-based speech synthesis method, employing the following technical solution:

[0005] Obtain the bit sequence of the input text, and use the ByT5 model to convert the bit sequence into a text feature vector;

[0006] The part-of-speech information and syntactic structure information of the input text are obtained, and information embedding encoding is performed on the part-of-speech information and syntactic structure information to obtain part-of-speech embedding vector and syntactic structure embedding vector;

[0007] The text feature vector, part-of-speech embedding vector, and syntactic structure embedding vector are summed and then input into the fully connected layer of the generative adversarial network speech model for linear operation to obtain the text output vector. The text output vector is then input into the activation function for feature fusion through nonlinear operation to obtain the text fusion features.

[0008] The text fusion features are input into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum. The target text MEL spectrum is then input into a vocoder for speech synthesis to obtain the target speech waveform.

[0009] Furthermore, after obtaining the text fusion features, and before inputting the text fusion features into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, the method further includes:

[0010] The text fusion features are input into the self-attention mechanism module of the generative adversarial network speech model;

[0011] The text enhancement features of the target text in the text fusion features are obtained using the self-attention mechanism module. The text enhancement features include semantic features and positional information of the text structure.

[0012] The step of inputting the text fusion features into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum includes:

[0013] The text fusion features and the text enhancement features are input into the decoder of the generative adversarial network speech model to obtain the MEL spectrum of the target text.

[0014] Furthermore, after inputting the text fusion features into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, and inputting the target text MEL spectrum into a vocoder for speech synthesis to obtain the target speech waveform, the method further includes:

[0015] The acquired target language Mel spectrum and the real target language Mel spectrum are input into the discriminator to calculate the adversarial loss information of the generative adversarial network speech model;

[0016] The parameters of the discriminator and the decoder are updated using the adversarial loss information.

[0017] Furthermore, obtaining the bit sequence of the input text includes:

[0018] Obtain the input text, encode the input text into characters, and obtain the numeric encoding value corresponding to each character in the input text;

[0019] Convert the numeric encoding value corresponding to each character into a binary number;

[0020] The binary numbers are concatenated in sequence to obtain the bit sequence of the input text.

[0021] Furthermore, the step of converting the bit sequence into a text feature vector using the ByT5 model includes:

[0022] The bit sequence of the input text is input into the input layer of the ByT5 model for preprocessing to obtain the normalized text bit sequence;

[0023] The standard text bit sequence is input into the embedding layer of the ByT5 model, and the embedding layer maps the standard text bit sequence into a text feature vector.

[0024] Furthermore, obtaining the part-of-speech information and syntactic structure information of the input text includes:

[0025] Get the HanLP part-of-speech tagging tool;

[0026] Use the HanLP part-of-speech tagging tool to create a part-of-speech tagger and a syntactic parser;

[0027] The part-of-speech tagger is used to obtain the part-of-speech information of each word in the input text, and the syntactic analyzer is used to obtain the syntactic structure information of the input text.

[0028] Furthermore, the creation of a part-of-speech tagger and a syntactic parser using the HanLP part-of-speech tagging tool includes:

[0029] Import the classes corresponding to the part-of-speech tagger and the syntactic analyzer into the HanLP part-of-speech tagging tool;

[0030] The HanLP part-of-speech tagging tool calls the class corresponding to the part-of-speech tagger to create a part-of-speech tagger, and uses the part-of-speech tagger to output each word and its corresponding part of speech. The HanLP part-of-speech tagging tool calls the class corresponding to the syntactic analyzer to create a syntactic analyzer, and uses the syntactic analyzer to output the syntactic analysis results.

[0031] To address the aforementioned problems, a ByT5-based speech synthesis device is also provided, employing the ByT5-based speech synthesis method, characterized by comprising:

[0032] The conversion module is used to obtain the bit sequence of the input text and convert the bit sequence into a text feature vector using the ByT5 model;

[0033] The information embedding module is used to obtain the part-of-speech information and syntactic structure information of the input text, perform information embedding encoding on the part-of-speech information and syntactic structure information, and obtain part-of-speech embedding vector and syntactic structure embedding vector;

[0034] The feature fusion module is used to sum the text feature vector, part-of-speech embedding vector, and syntactic structure embedding vector and input them into the fully connected layer of the generative adversarial network speech model for linear operation to obtain a text output vector. The text output vector is then input into an activation function for feature fusion through nonlinear operation to obtain the text fusion features.

[0035] The synthesis module is used to input the text fusion features into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, and input the target text MEL spectrum into the vocoder for speech synthesis to obtain the target speech waveform.

[0036] To address the aforementioned issues, this application also proposes a computer device, including a memory and a processor. The memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of a ByT5-based speech synthesis method.

[0037] To address the aforementioned issues, this application also proposes a computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of a ByT5-based speech synthesis method.

[0038] Compared with existing technologies, this method obtains the bit sequence of the input text, converts the bit sequence into a text feature vector using the ByT5 model, and then sums the text feature vector, part-of-speech embedding vector, and syntactic structure embedding vector before sequentially inputting them into a generative adversarial network speech model for feature fusion and decoding. This solves the problem of the dictionary limitation of token-based models, and can directly convert the original text into a unified bit representation for multiple languages. It eliminates the complex and lengthy text preprocessing process, which can greatly reduce error accumulation and amortize the cost of operating directly on the original text. By integrating syntactic and structural information based on the powerful general processing capabilities of ByT5, it can effectively improve the quality, naturalness, and prosody of synthesized speech. Attached Figure Description

[0039] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 A flowchart of an embodiment of a ByT5-based speech synthesis method according to this application;

[0041] Figure 2 yes Figure 1 A flowchart of the sub-steps of S100;

[0042] Figure 3 yes Figure 2 Flowchart following S130;

[0043] Figure 4 yes Figure 1 A flowchart of a specific implementation of S200;

[0044] Figure 5 yes Figure 1 A flowchart of a specific implementation of S220;

[0045] Figure 6 yes Figure 1 A flowchart of one embodiment following S300;

[0046] Figure 7 yes Figure 1 A flowchart of one embodiment following S400;

[0047] Figure 8 A flowchart illustrating a speech synthesis method based on ByT5 according to this application;

[0048] Figure 9 A schematic diagram of the module structure of an embodiment of a ByT5-based speech synthesis device according to this application;

[0049] Figure 10 A schematic diagram of the modular structure of a computer device according to this application. Detailed Implementation

[0050] The technical solutions of this invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are all within the scope of protection of this invention.

[0051] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings of this application are intended to cover non-exclusive inclusion.

[0053] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0054] The purpose of this application is to propose a speech synthesis method, apparatus, device, and storage medium based on ByT5, so as to solve the problems of existing token-based speech synthesis schemes, such as the limitation of the dictionary, poor speech effect, and semantic loss in the token-based model.

[0055] To solve the above technical problems, such as Figure 1 , Figure 1 A flowchart of a first embodiment of a ByT5-based speech synthesis method according to this application; this application provides a ByT5-based speech synthesis method, which adopts the following technical solution, including:

[0056] S100. Obtain the bit sequence of the input text and use the ByT5 model to convert the bit sequence into a text feature vector.

[0057] The ByT5-based speech synthesis method in this application runs on a computer device, whose communication module can be wired or wireless. It should be noted that the aforementioned wireless connection methods may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultrawideband) connections, and other currently known or future wireless connection methods.

[0058] The ByT5 model is an extension of the T5 (Text-to-Text Transfer Transformer) architecture. It employs byte-level encoding, allowing it to directly process raw text data without tokenization, unlike traditional token-based models. This design enables the model to learn and infer in a way that more closely resembles the original form of natural language, resulting in better performance and robustness when handling text in multiple languages ​​and formats. At its core is a Transformer-based Encoder-Decoder architecture that captures long-distance dependencies in text through self-attention and positional encoding. Compared to traditional T5 models, ByT5 adjusts its input and output layers to accommodate byte-level encoded data, giving it the potential for multilingual cross-language speech synthesis—an advantage over typical cross-language speech synthesis systems that can only perform one-to-one conversions.

[0059] The ByT5 model transforms all language tasks into unified, text-to-text tasks, allowing users to perform corresponding tasks simply by inputting text, as shown in Table 1.

[0060] Table 1

[0061]

[0062]

[0063] Table 1 shows examples of input text synthesis using the ByT5 model. As can be seen from Table 1, the ByT5 model possesses excellent cross-language processing capabilities and unified task capabilities. In this embodiment, the ByT5 model is used to specify cross-language speech synthesis tasks for different languages ​​by modifying the prefixes of the input text.

[0064] In some alternative implementations, such as Figure 2 , Figure 2 yes Figure 1 The flowchart of the sub-step S100 is as follows: S100 includes: S110, obtaining the input text, encoding the input text into characters, and obtaining the numeric encoding value corresponding to each character in the input text; S120, converting the numeric encoding value corresponding to each character into a binary number; S130, concatenating the binary numbers in order to obtain the bit sequence of the input text.

[0065] In this embodiment, the input text is stored in the computer device in the form of bytes. Any text and word phrase can be represented by bits or bit sequences. The ByT5 architecture, which uses byte encoding to obtain bit sequences and then represents words, has shown outstanding superiority in solving multilingual and unfamiliar word problems. Compared with general models that process at the token level, bit-level ByT5 is more robust to noise and performs better on word-level tasks that are sensitive to spelling and pronunciation.

[0066] For example, when dealing with noisy data, you can intentionally add noise to the input text data, such as random character errors or spelling mistakes. Compared to other models, the ByT5 model framework can more accurately understand and process this noisy data, and still generate relatively reasonable and accurate output, while other models may be significantly affected by noise, leading to worse output results.

[0067] For example, when the input text undergoes adversarial modifications, such as deliberately replacing key words or changing the word order, in an attempt to mislead the model, the ByT5 model framework can still correctly understand the meaning of the text and provide a reasonable response in such situations.

[0068] In some alternative implementations, such as Figure 3 , Figure 3 yes Figure 2 The flowchart after S130; after S130, S100 also includes: S140, inputting the bit sequence of the input text into the input layer of the ByT5 model for preprocessing to obtain the normalized text bit sequence; S150, inputting the normalized text bit sequence into the embedding layer of the ByT5 model, and mapping the normalized text bit sequence into a text feature vector through the embedding layer.

[0069] The ByT5 model's embedding layers consist of an input layer, an encoding layer, a decoding layer, and an output layer. The input layer handles byte-level input and positional encoding: to enable the model to distinguish bytes at different positions, the encoding layer adds positional encoding to each byte. The encoding layer is composed of multiple identical layers stacked together. Each encoding layer contains two main components: a multi-head self-attention mechanism and a feedforward neural network.

[0070] The decoding layer also consists of multiple layers, with a structure similar to the encoder layer, but each layer also contains an encoder-decoder attention mechanism. The output layer is used to generate the target text vector.

[0071] In this embodiment, the preprocessing includes digital filtering and normalization processing. The normalization processing includes padding or truncation, so that the bit sequence reaches a preset fixed length. By preprocessing the input bit sequence, digital noise can be filtered out, such as removing bit errors that occur during text transmission, and the bit sequence becomes more normalized.

[0072] S200. Obtain part-of-speech information and syntactic structure information of the input text, perform information embedding encoding on the part-of-speech information and syntactic structure information, and obtain part-of-speech embedding vector and syntactic structure embedding vector.

[0073] like Figure 8 , Figure 8 According to the flowchart of a speech synthesis method based on ByT5 in this application, the basic process is as follows:

[0074] The input text is converted into a bit sequence, and the ByT5 model is used to convert the bit sequence into a text feature vector.

[0075] The HanLP part-of-speech tagging tool is used to obtain part-of-speech information and syntactic structure information of the input text. The part-of-speech information and syntactic structure information are then embedded and encoded to obtain part-of-speech embedding vectors and syntactic structure embedding vectors.

[0076] The text feature vector, part-of-speech embedding vector, and syntactic structure embedding vector are summed and then input into a generative adversarial network speech model for processing to generate the target speech waveform.

[0077] The anti-network speech model includes a fully connected layer, an activation function layer, a self-attention mechanism module, a decoder, a vocoder, and a discriminator. The specific processing steps of the anti-network speech model will be described in detail later.

[0078] In some alternative implementations, such as Figure 4 , Figure 4 yes Figure 1 A flowchart of a specific implementation of S200; S200 includes: S210, obtaining the HanLP part-of-speech tagging tool; S220, creating a part-of-speech tagger and a syntactic analyzer using the HanLP part-of-speech tagging tool; S230, obtaining the part-of-speech information of each word in the input text using the part-of-speech tagger, and obtaining the syntactic structure information of the input text using the syntactic analyzer.

[0079] HanLP's syntactic analyzer, by comprehensively applying statistical machine learning methods and rules, can accurately tag the parts of speech of words in Chinese text.

[0080] In some alternative implementations, such as Figure 5 , Figure 5 yes Figure 1The flowchart illustrates a specific implementation of S220. S220 includes: S221, importing the classes corresponding to the part-of-speech tagger and the syntactic analyzer into the HanLP part-of-speech tagging tool; S222, the HanLP part-of-speech tagging tool calls the class corresponding to the part-of-speech tagger to create the part-of-speech tagger, outputs each word and its corresponding part of speech using the part-of-speech tagger, and calls the class corresponding to the syntactic analyzer to create the syntactic analyzer, outputting the syntactic analysis results using the syntactic analyzer. HanLP (Han Language Processing) is a Chinese language processing toolkit. The HanLP tool uses the created part-of-speech tagger object to process text and obtain the part-of-speech information of each word. HanLP's part-of-speech tagging tool can accurately tag the parts of speech of words in Chinese text by comprehensively applying statistical machine learning methods and rules.

[0081] S300: The text feature vector, part-of-speech embedding vector, and syntactic structure embedding vector are summed and then input into the fully connected layer of the generative adversarial network speech model for linear operation to obtain the text output vector. The text output vector is then input into the activation function for feature fusion through nonlinear operation to obtain the text fusion features.

[0082] In this embodiment, the fully connected layer integrates different feature vectors through linear operations. The fully connected layer connects all the input feature vectors, allowing each output neuron to receive information from the entire feature vector, thus enabling the fully connected layer to learn different feature information of the entire text. The activation function controls the output range through feature fusion, preparing for classification decisions.

[0083] In some alternative implementations, such as Figure 6 ,yes Figure 1 The flowchart of an embodiment following S300 is located after S300 and before S400. The method further includes: S310, inputting text fusion features into the self-attention mechanism module of the generative adversarial network speech model; S320, using the self-attention mechanism module to obtain text enhancement features of the target text in the text fusion features, the text enhancement features including semantic features and positional information of text structure; S330, inputting the text fusion features into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, including: inputting the text fusion features and text enhancement features into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum.

[0084] By employing a self-attention mechanism module, the anti-network speech model is able to focus on important information in the fused features, as well as the positional information of the text's semantics and structure.

[0085] The self-attention mechanism module mainly enables anti-network speech models to focus on the relationships between different positions in a sequence when processing sequential data (such as text), thereby better understanding the semantic information of the entire sequence.

[0086] For an input sequence, the self-attention mechanism first transforms each element (e.g., a word vector) into three vectors through a linear transformation: a query vector, a key vector, and a value vector. Then, for each position in the sequence, it calculates the dot product of the query vector at that position with the key vectors of all other positions. After scaling and other processing, it uses a softmax function to obtain the attention weight. This weight represents the degree of correlation between the current position and other positions. Finally, it performs a weighted sum of the attention weights and the corresponding value vectors to obtain the output for that position. This output incorporates information from other positions in the sequence and assigns different levels of importance to the information at different positions based on the magnitude of the attention weights.

[0087] By employing a self-attention mechanism, dependencies between any two positions in a sequence can be effectively captured, regardless of their distance. This allows generative adversarial network (GAN) speech models to better understand global semantic information and avoids the problem of capturing long-distance dependencies in the model.

[0088] S400. Input the text fusion features into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum. Input the target text MEL spectrum into the vocoder for speech synthesis to obtain the target speech waveform.

[0089] In this embodiment, the decoder includes a function that generates continuous values, mainly used to generate the Mel spectrum. The number of neurons in the decoder output layer corresponds to the dimension of the Mel spectrum.

[0090] In this embodiment, the vocoder is used to select a suitable excitation signal and modulate the excitation signal so that the energy distribution of the excitation signal is adjusted according to the envelope information. The modulated excitation signal is then passed through a bandpass filter to reconstruct an approximation of the original speech signal, that is, to generate the target speech waveform, thereby generating natural and realistic synthesized speech.

[0091] Because the added features may contain noise, in order to enable the model to generate high-quality speech, in this embodiment, an adversarial generative training strategy is adopted after the Mel spectrum decoder. The discriminator is used to calculate the loss, and then the discriminator parameters and the parameters of the decoder generating the Mel spectrum are updated based on the calculation results. In step S400, when using the self-attention mechanism module to obtain the text enhancement features of the target text from the text fusion features, the text fusion features and text enhancement features are input into the decoder of the generative adversarial network speech model to obtain the target text Mel spectrum.

[0092] Furthermore, such as Figure 7 ,yes Figure 1 The flowchart of an embodiment following S400 shows that, after S400, the method further includes: S500, inputting the acquired target language Mel spectrum and the real target language Mel spectrum into the discriminator to calculate the adversarial loss information of the generated adversarial network speech model; S600, updating the parameters of the discriminator and the decoder using the adversarial loss information.

[0093] In this embodiment, an adversarial training strategy is adopted after the Mel spectrum decoder. By using the discriminator to calculate the loss, the discriminator parameters and the parameters of the decoder to generate the Mel spectrum are updated according to the calculation results. This enables the generative adversarial network speech model to still generate high-quality target speech in input text containing noise.

[0094] During training, a large amount of target language speech data is collected through various methods. This can be obtained from publicly available speech databases such as TIMIT and LibriSpeech, which typically contain high-quality speech recordings in multiple languages. Alternatively, speech data can be recorded independently, for example, by inviting native speakers of the target language to record readings or dialogues. Then, Fourier transforms and Mel-spectrum calculations are performed on the collected speech samples to obtain the true Mel-spectrum for the target language. These Mel-spectrums can be used as training data to train speech recognition, speech synthesis, and other models, thereby improving the model's performance and accuracy in the target language.

[0095] For example, suppose there is a speech synthesis task with the goal of generating natural and fluent speech, while a discriminator judges whether the generated speech is authentic. Generative adversarial network (GAN) speech models, when generating speech waveforms, make subtle perturbations to certain key audio features. For example, they randomly adjust features such as the fundamental frequency and formants of the speech within a certain range.

[0096] For example, for a specific statement, the generator might slightly raise or lower its fundamental frequency, making the sound slightly different, but the overall semantics and expression remain unchanged. The generator can also attempt to blend multiple different speech styles or features to confuse the discriminator. It can combine the timbre characteristics, speech rate variations, and emotional inclinations of different speakers to generate a mixed output.

[0097] Through these adversarial strategies, the generator continuously attempts to break through the discriminator's judgment, which can gradually improve the quality and authenticity of the generated speech in the entire generative adversarial network speech model (GAN model) during the adversarial process.

[0098] To solve the above problems, such as Figure 9 , Figure 9A schematic diagram of the module structure of a ByT5-based speech synthesis device 10 according to this application; embodiments of this application also propose a ByT5-based speech synthesis device 700, employing a ByT5-based speech synthesis method, including:

[0099] The conversion module 701 is used to obtain the bit sequence of the input text and convert the bit sequence into a text feature vector using the ByT5 model;

[0100] The information embedding module 702 is used to obtain part-of-speech information and syntactic structure information of the input text, perform information embedding encoding on the part-of-speech information and syntactic structure information, and obtain part-of-speech embedding vector and syntactic structure embedding vector;

[0101] The feature fusion module 703 is used to sum the text feature vector, part-of-speech embedding vector, and syntactic structure embedding vector and input them into the fully connected layer of the generative adversarial network speech model for linear operation to obtain the text output vector. The text output vector is then input into the activation function for feature fusion through nonlinear operation to obtain the text fusion features.

[0102] The synthesis module 704 is used to input the text fusion features into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, and input the target text MEL spectrum into the vocoder for speech synthesis to obtain the target speech waveform.

[0103] By acquiring the bit sequence of the input text, the ByT5 model is used to convert the bit sequence into a text feature vector. The text feature vector, part-of-speech embedding vector, and syntactic structure embedding vector are summed and then sequentially input into a generative adversarial network speech model for feature fusion and decoding. This solves the problem of the dictionary limitation of token-based models, and can directly convert the original text into a unified bit representation for multiple languages. It removes the complex and lengthy text preprocessing process, which can greatly reduce error accumulation and amortize the cost of operating directly on the original text. Based on the powerful general processing capabilities of ByT5, it integrates syntactic and structural information, which can effectively improve the quality, naturalness and prosody of synthesized speech.

[0104] To solve the above problems, such as Figure 10 , Figure 10 A schematic diagram of the module structure of a computer device according to this application. Embodiments of this application also propose a computer device including a memory and a processor. The memory stores computer-readable instructions, and the processor, when executing the computer-readable instructions, implements the steps of a ByT5-based speech synthesis method.

[0105] To address the aforementioned issues, this application also proposes a computer-readable storage medium storing computer-readable instructions. When these instructions are executed by a processor, they implement the steps of a ByT5-based speech synthesis method.

[0106] To address the aforementioned issues, this application also proposes a computer device that employs the following technical solution: it includes a memory, a communication module, and a processor. The memory stores computer-readable instructions, and the processor executes these computer-readable instructions to implement the steps of a ByT5-based speech synthesis method.

[0107] This computer device can be a computer, server, workstation, or other similar device; it can also be a mobile device such as a mobile phone, tablet, or in-vehicle mobile terminal; or other devices with program execution capabilities. Its internal structure diagram can be as follows: Figure 10 As shown, Figure 10 This is a schematic diagram of a computer device according to an embodiment of the present application. The computer device includes a processor, a memory, and a communication module. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores an operating system, instructions, or code. The internal memory provides an environment for the operation of the operating system and instructions or code in the non-volatile storage media. When the instructions or code are executed by the processor, they implement the functions or steps of a fine-tuning method for a pre-trained model as described above. The communication module of the computer device may include a network interface and / or a wireless communication module, allowing the computer device to communicate with other devices or service platforms. Furthermore, the computer device may also include a display screen and input devices, etc.

[0108] The memory stores computer programs, which include program instructions. The processor is configured to call these instructions. When the processor executes the instructions or code, it performs the following steps: S100: Obtain the bit sequence of the input text and convert it into a text feature vector using the ByT5 model; S200: Obtain the part-of-speech (POS) and syntactic structure (SHS) information of the input text, perform information embedding encoding on the POS and SHS information, and obtain POS embedding vectors and SHS embedding vectors; S300: Summate the text feature vectors, POS embedding vectors, and SHS embedding vectors and input them into the fully connected layer of the generative adversarial network (GAN) speech model for linear operation to obtain the text output vector. Input the text output vector into the activation function for feature fusion through nonlinear operation to obtain the text fusion feature; S400: Input the text fusion feature into the decoder of the GAN speech model to obtain the target text MEL spectrum. Input the target text MEL spectrum into the vocoder for speech synthesis to obtain the target speech waveform.

[0109] To address the aforementioned problems, this application also proposes a computer-readable storage medium, employing the following technical solution:

[0110] A computer-readable storage medium stores a computer program, which includes program instructions that, when executed by a processor, cause the processor to perform an adaptive parameter-efficient fine-tuning method.

[0111] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which are implemented when executed by a processor. Figures 1 to 8 The steps described above provide a speech synthesis method based on ByT5, and the specific implementation details are provided in each step. These details will not be repeated here.

[0112] The aforementioned computer-readable storage medium may be a standardized lip-sync speaking face video generation device provided in any of the foregoing embodiments, or an internal storage unit of the aforementioned terminal device, such as a hard drive or memory of a computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device.

[0113] Furthermore, the computer-readable storage medium may include both internal storage units and external storage devices of the computer device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0114] However, it should be understood that it is not required to implement all the components shown; more or fewer components may be implemented instead. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0115] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0116] Compared with existing technologies, this method obtains the bit sequence of the input text, uses the ByT5 model to convert the bit sequence into a text feature vector, and then sums the text feature vector, part-of-speech embedding vector, and syntactic structure embedding vector before feeding them into a generative adversarial network speech model for feature fusion and decoding. This solves the problem of the dictionary limitation of token-based models, and can directly convert the original text into a unified bit representation for multiple languages. It eliminates the complex and lengthy text preprocessing process, which can greatly reduce error accumulation and amortize the cost of operating directly on the original text. By integrating syntactic and structural information based on the powerful general processing capabilities of ByT5, it can effectively improve the quality, naturalness, and prosody of synthesized speech.

[0117] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0118] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A speech synthesis method based on ByT5, characterized in that it includes: The input text bit sequence is obtained, and the bit sequence is converted into a text feature vector using the ByT5 model. The core of the ByT5 model is an Encoder-Decoder architecture based on Transformer. The ByT5 model captures long-distance dependencies in the text through self-attention mechanism and positional encoding. The ByT5 model is an extension based on the Text-to-TextTransfer Transformer architecture and adopts byte-level encoding. The part-of-speech information and syntactic structure information of the input text are obtained, and information embedding encoding is performed on the part-of-speech information and syntactic structure information to obtain part-of-speech embedding vector and syntactic structure embedding vector; The text feature vector, part-of-speech embedding vector, and syntactic structure embedding vector are summed and then input into the fully connected layer of the generative adversarial network speech model for linear operation to obtain the text output vector. The text output vector is then input into the activation function for feature fusion through nonlinear operation to obtain the text fusion features. The text fusion features are input into the decoder of the generative adversarial network speech model to obtain the target speech MEL spectrum. The target speech MEL spectrum is then input into a vocoder for speech synthesis to obtain the target language waveform.

2. The speech synthesis method based on ByT5 according to claim 1, characterized in that, After obtaining the text fusion features, and before inputting the text fusion features into the decoder of the generative adversarial network speech model to obtain the target speech MEL spectrum, the method further includes: The text fusion features are input into the self-attention mechanism module of the generative adversarial network speech model; The text enhancement features of the target text in the text fusion features are obtained using the self-attention mechanism module. The text enhancement features include semantic features and positional information of the text structure. The step of inputting the text fusion features into the decoder of the generative adversarial network speech model to obtain the target speech MEL spectrum includes: The text fusion features and the text enhancement features are input into the decoder of the generative adversarial network speech model to obtain the target speech MEL spectrum.

3. The speech synthesis method based on ByT5 according to claim 2, characterized in that, The method further includes inputting the text fusion features into the decoder of the generative adversarial network speech model to obtain the target speech MEL spectrum, inputting the target speech MEL spectrum into a vocoder for speech synthesis to obtain the target speech waveform, and then: The acquired target speech MEL spectrum and the real target language MEL spectrum are input into the discriminator to calculate the adversarial loss information of the generative adversarial network speech model; The parameters of the discriminator and the decoder are updated using the adversarial loss information.

4. The speech synthesis method based on ByT5 according to claim 1, characterized in that, The step of obtaining the bit sequence of the input text includes: Obtain the input text, encode the input text into characters, and obtain the numeric encoding value corresponding to each character in the input text; Convert the numeric encoding value corresponding to each character into a binary number; The binary numbers are concatenated in sequence to obtain the bit sequence of the input text.

5. The speech synthesis method based on ByT5 according to claim 4, characterized in that, The process of converting the bit sequence into a text feature vector using the ByT5 model includes: The bit sequence of the input text is input into the input layer of the ByT5 model for preprocessing to obtain the normalized text bit sequence; The standard text bit sequence is input into the embedding layer of the ByT5 model, and the embedding layer maps the standard text bit sequence into a text feature vector.

6. The speech synthesis method based on ByT5 according to claim 1, characterized in that, The step of obtaining the part-of-speech information and syntactic structure information of the input text includes: Get the HanLP part-of-speech tagging tool; Use the HanLP part-of-speech tagging tool to create a part-of-speech tagger and a syntactic parser; The part-of-speech tagger is used to obtain the part-of-speech information of each word in the input text, and the syntactic analyzer is used to obtain the syntactic structure information of the input text.

7. The speech synthesis method based on ByT5 according to claim 6, characterized in that, The process of creating a part-of-speech tagger and a syntactic parser using the HanLP part-of-speech tagging tool includes: Import the classes corresponding to the part-of-speech tagger and the syntactic analyzer into the HanLP part-of-speech tagging tool; The HanLP part-of-speech tagging tool calls the class corresponding to the part-of-speech tagger to create a part-of-speech tagger, and uses the part-of-speech tagger to output each word and its corresponding part of speech. The HanLP part-of-speech tagging tool calls the class corresponding to the syntactic analyzer to create a syntactic analyzer, and uses the syntactic analyzer to output the syntactic analysis results.

8. A ByT5-based speech synthesis device, employing the ByT5-based speech synthesis method according to any one of claims 1-7, characterized in that, include: The conversion module is used to obtain the bit sequence of the input text and convert the bit sequence into a text feature vector using the ByT5 model. The core of the ByT5 model is an Encoder-Decoder architecture based on Transformer. The ByT5 model captures long-distance dependencies in the text through self-attention mechanism and positional encoding. The ByT5 model is an extension based on the Text-to-Text Transfer Transformer architecture and adopts a byte-level encoding method. The information embedding module is used to obtain the part-of-speech information and syntactic structure information of the input text, perform information embedding encoding on the part-of-speech information and syntactic structure information, and obtain part-of-speech embedding vector and syntactic structure embedding vector; The feature fusion module is used to sum the text feature vector, part-of-speech embedding vector, and syntactic structure embedding vector and input them into the fully connected layer of the generative adversarial network speech model for linear operation to obtain a text output vector. The text output vector is then input into an activation function for feature fusion through nonlinear operation to obtain the text fusion features. The synthesis module is used to input the text fusion features into the decoder of the generative adversarial network speech model to obtain the target speech MEL spectrum, and input the target speech MEL spectrum into the vocoder for speech synthesis to obtain the target language waveform.

9. A computer device, characterized in that, It includes a memory and a processor, the memory storing computer-readable instructions, and the processor executing the steps of the ByT5-based speech synthesis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the ByT5-based speech synthesis method as described in any one of claims 1 to 7.

Images