Speech synthesis method and device based on ByT5, equipment and storage medium

Abstract

The invention belongs to the technical field of speech synthesis, and discloses a ByT5-based speech synthesis method, device and equipment and a storage medium, and the method comprises the steps: obtaining a bit sequence of an input text, and converting the bit sequence into a text feature vector through a ByT5 model; obtaining a part-of-speech embedding vector and a syntactic structure embedding vector; summing the text feature vector, the part-of-speech embedded vector and the syntactic structure embedded vector, then sequentially inputting the summed text feature vector, the part-of-speech embedded vector and the syntactic structure embedded vector into a full connection layer of a generative adversarial network speech model, and activating a function for feature fusion to obtain text fusion features; and a decoder and a vocoder of the generative adversarial network voice model are used to successively acquire a target language Me < l > spectrum and a target language waveform. According to the method, the problem that a token-based model is limited by a dictionary is solved, an original text can be directly converted to multilingual unified bit representation, syntax and structure information is fused on the basis of the strong general processing capacity of a ByT5 model framework, and the quality of synthesized speech can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis technology, and in particular to a speech synthesis method, device, equipment and storage medium based on ByT5. Background Art

[0002] Cross-language speech synthesis tasks involve converting speech text in one language into natural and fluent speech text in another language. This task has important application prospects in multilingual society and international applications, and can improve the user experience of speech interaction systems while providing a wider range of services. For example, remote consultation: In remote medical consultation, doctors need to communicate about the patient's condition and diagnosis. Speech synthesis technology can convert the text information entered by doctors into speech, facilitating communication between doctors. For example, during remote consultation, experts can use speech synthesis technology to convey their diagnosis and treatment suggestions to other doctors, improving the efficiency and accuracy of the consultation. Remote monitoring: For some patients with chronic diseases or patients recovering at home, the remote monitoring system can use speech synthesis technology to regularly send voice reminders to patients, such as reminding patients to take medicine on time, measure blood pressure, blood sugar, etc., and at the same time feed back the patient's monitoring data to the doctor, so that the doctor can understand the patient's condition changes in a timely manner. At present, most text-to-speech (TTS) systems need to work based on tokens in the text processing stage. The defects of this technical solution are: the speech synthesis model can only recognize and process characters within the vocabulary (that is, there are corresponding tokens in the vocabulary), and the model is often unable to process unknown words. In addition, the speech synthesized by existing cross-language speech synthesis methods is often not good, and there are problems such as semantic loss. Summary of the invention

[0003] The purpose of the embodiments of the present application is to propose a speech synthesis method, apparatus, device and storage medium based on ByT5 to solve the problems in existing token-based speech synthesis solutions that the token-based model is subject to dictionary restrictions, poor speech effects and semantic loss.

[0004] In order to solve the above technical problems, the embodiment of the present application provides a speech synthesis method based on ByT5, which adopts the following technical solution, including:

[0005] Obtain a bit sequence of an input text, and convert the bit sequence into a text feature vector using a ByT5 model;

[0006] Obtaining part-of-speech information and syntactic structure information of the input text, performing information embedding encoding on the part-of-speech information and syntactic structure information, and obtaining a part-of-speech embedding vector and a syntactic structure embedding vector;

[0007] The text feature vector, the part-of-speech embedding vector, and the syntactic structure embedding vector are summed and input into the fully connected layer of the generative adversarial network speech model for linear operation to obtain a text output vector, and the text output vector is input into an activation function for feature fusion through nonlinear operation to obtain a text fusion feature;

[0008] The text fusion feature is input into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, and the target text MEL spectrum is input into the vocoder for speech synthesis to obtain the target speech waveform.

[0009] Further, after obtaining the text fusion feature, before inputting the text fusion feature into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, the method also includes:

[0010] Inputting the text fusion features into the self-attention mechanism module of the generative adversarial network speech model;

[0011] Using the self-attention mechanism module to obtain text enhancement features of the target text in the text fusion features, the text enhancement features include semantic features and position information of the text structure;

[0012] The step of inputting the text fusion feature into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum includes:

[0013] The text fusion feature and the text enhancement feature are input into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum.

[0014] Further, after inputting the text fusion feature into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, inputting the target text MEL spectrum into the vocoder for speech synthesis to obtain the target speech waveform, the method further includes:

[0015] Input the acquired target language Mel spectrum and the real target language Mel spectrum into the discriminator, and calculate the adversarial loss information of the generative adversarial network speech model;

[0016] The adversarial loss information is used to update parameters of the discriminator and the decoder.

[0017] Further, the step of obtaining the bit sequence of the input text includes:

[0018] Obtaining input text, performing character encoding on the input text, and obtaining a digital encoding value corresponding to each character in the input text;

[0019] Convert the digital code value corresponding to each character into a binary number;

[0020] The binary numbers are connected in sequence to obtain a bit sequence of the input text.

[0021] Furthermore, the step of converting the bit sequence into a text feature vector using the ByT5 model includes:

[0022] Inputting the bit sequence of the input text into the input layer of the ByT5 model for preprocessing to obtain a standard text bit sequence;

[0023] The standard text bit sequence is input into the embedding layer of the ByT5 model, and the standard text bit sequence is mapped into a text feature vector through the embedding layer.

[0024] Furthermore, the obtaining of part-of-speech information and syntactic structure information of the input text includes:

[0025] Get the HanLP part-of-speech tagging tool;

[0026] Using the HanLP part-of-speech tagging tool to create a part-of-speech tagger and a syntactic analyzer;

[0027] The part-of-speech tagger is used to obtain the part-of-speech information of each word in the input text, and the syntactic analyzer is used to obtain the syntactic structure information of the input text.

[0028] Further, the use of the HanLP part-of-speech tagging tool to create a part-of-speech tagger and a syntactic analyzer includes:

[0029] Import the classes corresponding to the part-of-speech tagger and syntactic analyzer into the HanLP part-of-speech tagging tool;

[0030] The HanLP part-of-speech tagging tool calls the class corresponding to the part-of-speech tagger to create a part-of-speech tagger, and uses the part-of-speech tagger to output each word and its corresponding part of speech. The HanLP part-of-speech tagging tool calls the class corresponding to the syntactic analyzer to create a syntactic analyzer, and uses the syntactic analyzer to output a syntactic analysis result.

[0031] In order to solve the above problem, a speech synthesis device based on ByT5 is also provided, which adopts the speech synthesis method based on ByT5, and is characterized by comprising:

[0032] A conversion module, used for obtaining a bit sequence of an input text, and converting the bit sequence into a text feature vector using a ByT5 model;

[0033] An information embedding module, used to obtain the part-of-speech information and syntactic structure information of the input text, perform information embedding encoding on the part-of-speech information and syntactic structure information, and obtain a part-of-speech embedding vector and a syntactic structure embedding vector;

[0034] A feature fusion module is used to sum the text feature vector, the part-of-speech embedding vector, and the syntactic structure embedding vector and input them into the fully connected layer of the generative adversarial network speech model for linear operation to obtain a text output vector, and input the text output vector into an activation function for feature fusion through nonlinear operation to obtain a text fusion feature;

[0035] A synthesis module is used to input the text fusion feature into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, input the target text MEL spectrum into the vocoder for speech synthesis, and obtain the target speech waveform.

[0036] In order to solve the above problems, an embodiment of the present application further proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of a speech synthesis method based on ByT5.

[0037] In order to solve the above problems, an embodiment of the present application further proposes a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the steps of a speech synthesis method based on ByT5 are implemented.

[0038] Compared with the prior art, the present invention obtains the bit sequence of the input text, converts the bit sequence into a text feature vector using the ByT5 model, sums the text feature vector, part-of-speech embedding vector, and syntactic structure embedding vector, and sequentially inputs them into the generative adversarial network speech model for feature fusion and decoding. This solves the problem that the token-based model is restricted by the dictionary, can directly convert the original text into a unified bit representation in multiple languages, removes the complex and lengthy text preprocessing process, can greatly reduce error accumulation, amortizes the cost of operating directly on the original text, and utilizes the powerful general processing capabilities of ByT5 to integrate syntactic and structural information, which can effectively improve the quality, naturalness and rhythm of the synthesized speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A flowchart of an embodiment of a speech synthesis method based on ByT5 according to the present application;

[0041] Figure 2 yes Figure 1 Flow chart of the sub-steps of S100;

[0042] Figure 3 yes Figure 2 Flowchart after S130;

[0043] Figure 4 yes Figure 1 A flowchart of a specific implementation of S200;

[0044] Figure 5 yes Figure 1 A flowchart of a specific implementation of S220;

[0045] Figure 6 yes Figure 1 A flowchart of an embodiment after S300;

[0046] Figure 7 yes Figure 1 A flowchart of an embodiment after S400;

[0047] Figure 8 A flowchart diagram of a speech synthesis method based on ByT5 according to the present application;

[0048] Fig. 9 A schematic diagram of the module structure of an embodiment of a speech synthesis device based on ByT5 according to the present application;

[0049] Fig.10 A schematic diagram of the module structure of a computer device according to the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the accompanying drawings in the invention to clearly and completely describe the technical solutions in the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the scope of protection of the present invention.

[0051] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used in the specification of the application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0053] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0054] The purpose of the embodiments of the present application is to propose a speech synthesis method, apparatus, device and storage medium based on ByT5 to solve the problems in existing token-based speech synthesis solutions that the token-based model is subject to dictionary restrictions, poor speech effects and semantic loss.

[0055] In order to solve the above technical problems, Figure 1 , Figure 1 According to a flowchart of a first embodiment of a speech synthesis method based on ByT5 of the present application; the embodiment of the present application provides a speech synthesis method based on ByT5, which adopts the following technical solution, including:

[0056] S100, obtaining a bit sequence of an input text, and converting the bit sequence into a text feature vector using a ByT5 model.

[0057] The speech synthesis method based on ByT5 in the embodiment of the present application runs on a computer device, and the communication module of the computer device can adopt a wired connection mode or a wireless connection mode. It should be noted that the above-mentioned wireless connection mode may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, Wi MAX connection, Zigbee connection, UWB (ultrawideband) connection, and other wireless connection modes currently known or to be developed in the future.

[0058] The ByT5 model is an extension of the T5 (Text-to-Text Transfer Transformer) architecture. It uses byte-level encoding. Compared with traditional token-based models, ByT5 can directly process raw text data without tokenization. This design enables the model to learn and reason in a way that is closer to the original form of natural language, thereby showing better performance and robustness when processing text in multiple languages ​​and formats. The core of the ByT5 model is the Transformer-based Encoder-Decoder architecture, which captures long-distance dependencies in text through self-attention and positional encoding. Compared with the traditional T5 model, ByT5 has been adjusted in the input and output layers to adapt to byte-level encoded data. The ByT5 model has the potential for cross-language speech synthesis in multiple languages, which is superior to general cross-language speech synthesis systems that can only perform one-to-one conversion.

[0059] The ByT5 model converts all language tasks into unified, text-to-text tasks, and the corresponding tasks can be performed by inputting text, as shown in Table 1.

[0060] Table 1

[0061]

[0062]

[0063] Table 1 is an example of synthesizing input text using the ByT5 model. As can be seen from Table 1, the ByT5 model has excellent cross-language processing capabilities and unified task capabilities. In the embodiment of the present application, a cross-language speech synthesis task in different languages ​​is specified by modifying the prefix of the input text based on the ByT5 model.

[0064] In some optional embodiments, such as Figure 2 , Figure 2 yes Figure 1 Flowchart of the sub-steps of S100; S100 includes: S110, obtaining input text, character encoding the input text, and obtaining the digital code value corresponding to each character in the input text; S120, converting the digital code value corresponding to each character into a binary number; S130, connecting the binary numbers in sequence to obtain a bit sequence of the input text.

[0065] In an embodiment of the present application, the input text is stored in a computer device in the form of bytes. Any text and vocabulary phrases can be represented by bits or bit sequences. The ByT5 architecture that uses byte encoding to obtain bit sequences and then represents characters has shown outstanding superiority in solving multilingual and new word problems. Compared with models that generally process at the token level, the bit-level ByT5 is more robust to noise and performs better on word-level tasks that are sensitive to spelling and pronunciation.

[0066] For example, when processing noisy data, some noise can be deliberately added to the input text data, such as random character errors, spelling errors, etc. Compared with other models, the ByT5 model framework can understand and process these noisy data more accurately and still generate more reasonable and accurate outputs, while other models may be greatly disturbed by noise, resulting in poor output results.

[0067] For example, when the input text is modified adversarially, such as deliberately replacing some key words or changing the order of words, in an attempt to mislead the model, the ByT5 model framework can still correctly understand the meaning of the text and give a reasonable response in this case.

[0068] In some optional embodiments, such as Figure 3 , Figure 3 yes Figure 2 The flowchart after S130; after S130, S100 also includes: S140, inputting the bit sequence of the input text into the input layer of the ByT5 model for preprocessing to obtain a standard text bit sequence; S150, inputting the standard text bit sequence into the embedding layer of the ByT5 model, and mapping the standard text bit sequence into a text feature vector through the embedding layer.

[0069] The embedding layer of the ByT5 model includes input layer, encoding layer, decoding layer and output layer. The input layer is used for byte-level input and position encoding: In order to enable the model to distinguish bytes at different positions, the encoding layer adds position encoding to each byte. The encoding layer is composed of multiple identical layers stacked together. Each encoding layer consists of two main parts: a multi-head self-attention mechanism and a feedforward neural network.

[0070] The decoding layer also consists of multiple layers with a similar structure to the encoder layer, but also contains an encoder-decoder attention mechanism in each layer. The output layer is used to generate the target text vector.

[0071] In this embodiment, preprocessing includes digital filtering and normalization processing, wherein the normalization processing includes padding or truncation so that the bit sequence reaches a preset fixed length. By preprocessing the input bit sequence, digital noise can be filtered out, for example, bit errors occurring during text transmission can be removed, and the bit sequence can be more normalized.

[0072] S200, obtaining part-of-speech information and syntactic structure information of the input text, performing information embedding encoding on the part-of-speech information and the syntactic structure information, and obtaining a part-of-speech embedding vector and a syntactic structure embedding vector.

[0073] like Figure 8 , Figure 8 According to a flowchart of a speech synthesis method based on ByT5 of the present application, the basic process is as follows:

[0074] Convert the input text into a bit sequence, and use the ByT5 model to convert the bit sequence into a text feature vector;

[0075] Use the HanLP part-of-speech tagging tool to obtain the part-of-speech information and syntactic structure information of the input text, perform information embedding encoding on the part-of-speech information and syntactic structure information, and obtain the part-of-speech embedding vector and syntactic structure embedding vector;

[0076] The text feature vector, part-of-speech embedding vector, and syntactic structure embedding vector are summed and input into the generative adversarial network speech model for processing to generate the target speech waveform.

[0077] Among them, the anti-network speech model includes a fully connected layer, an activation function layer, a self-attention mechanism module, a decoder, a vocoder and a discriminator. The specific processing process of the anti-network speech model will be introduced in detail later.

[0078] In some optional embodiments, such as Figure 4 , Figure 4 yes Figure 1 A flowchart of a specific implementation method of S200; S200 includes: S210, obtaining a HanLP part-of-speech tagging tool; S220, using the HanLP part-of-speech tagging tool to create a part-of-speech tagger and a syntactic analyzer; S230, using the part-of-speech tagger to obtain the part-of-speech information of each word in the input text, and using the syntactic analyzer to obtain the syntactic structure information of the input text.

[0079] HanLP's syntactic analyzer can accurately label the parts of speech of words in Chinese text by combining statistical machine learning methods and rules.

[0080] In some optional embodiments, such as Figure 5 , Figure 5 yes Figure 1A flowchart of a specific implementation of S220 in the figure; S220 includes: S221, importing the classes corresponding to the part-of-speech tagger and the syntactic analyzer into the HanLP part-of-speech tagging tool; S222, the HanLP part-of-speech tagging tool calls the class corresponding to the part-of-speech tagger to create a part-of-speech tagger, and uses the part-of-speech tagger to output each word and the corresponding part of speech, and the HanLP part-of-speech tagging tool calls the class corresponding to the syntactic analyzer to create a syntactic analyzer, and uses the syntactic analyzer to output the syntactic analysis result. HanLP (Han Language Processing) is a Chinese language processing toolkit. The HanLP tool uses the created part-of-speech tagger object to process the text and obtain the part-of-speech information of each word. The HanLP part-of-speech tagging tool can more accurately mark the part of speech for words in Chinese text by comprehensively applying statistical machine learning methods and rules.

[0081] S300, summing up the text feature vector, the part-of-speech embedding vector, and the syntactic structure embedding vector, and inputting the sum into the fully connected layer of the generative adversarial network speech model for linear operation to obtain a text output vector, and inputting the text output vector into an activation function for feature fusion through nonlinear operation to obtain text fusion features.

[0082] In this embodiment, the fully connected layer integrates different feature vectors through linear operations. The fully connected layer connects all input feature vectors so that each output neuron can receive the information of the entire feature vector, so that the fully connected layer can learn different feature information of the entire text. The activation function controls the output range through feature fusion to prepare for classification decisions.

[0083] In some optional embodiments, such as Figure 6 ,yes Figure 1 A flowchart of an embodiment after S300 is after S300 and before S400, and the method also includes: S310, inputting the text fusion feature into the self-attention mechanism module of the generative adversarial network speech model; S320, using the self-attention mechanism module to obtain the text enhancement feature of the target text in the text fusion feature, the text enhancement feature includes the semantic feature and the position information of the text structure; S330, inputting the text fusion feature into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, including: inputting the text fusion feature and the text enhancement feature into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum.

[0084] By adopting the self-attention mechanism module, the anti-network speech model can pay attention to the important information in the fused features, as well as the location information of the text semantics and structure.

[0085] The self-attention mechanism module mainly enables the anti-network speech model to focus on the relationship between different positions in the sequence when processing sequence data (such as text), so as to better understand the semantic information of the entire sequence.

[0086] For an input sequence, the self-attention mechanism module first transforms each element (such as a word vector) linearly to obtain three vectors: query vector (Query), key vector (Key) and value vector (Value). Then, for each position in the sequence, the dot product of the query vector at that position and the key vectors at all other positions is calculated, and after scaling and other processing, the attention weight is obtained through the softmax function. This weight represents the degree of association between the current position and other positions. Finally, the attention weight is weighted and summed with the value vector of the corresponding position to obtain the output of the position. This output incorporates the information of other positions in the sequence, and assigns different degrees of importance to the information at different positions according to the size of the attention weight.

[0087] By adopting the self-attention mechanism module, the dependency between any two positions in the sequence can be effectively captured, no matter how far apart they are. This enables the generative adversarial network speech model to better understand global semantic information and avoid the problem of long-distance dependencies being difficult to capture in the model.

[0088] S400, input the text fusion feature into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, input the target text MEL spectrum into the vocoder for speech synthesis, and obtain the target speech waveform.

[0089] In this embodiment, the decoder includes a function for generating continuous values, which is mainly used to generate a Mel spectrum, and the number of neurons in the output layer of the decoder corresponds to the dimension of the Mel spectrum.

[0090] In this embodiment, the vocoder is used to select a suitable excitation signal and modulate the excitation signal so that the energy distribution of the excitation signal is adjusted according to the envelope information. The modulated excitation signal is passed through a bandpass filter to reconstruct an approximation of the original speech signal, that is, to generate a target speech waveform, thereby generating natural and realistic synthesized speech.

[0091] Because the additional features may contain noise, in order to enable the model to generate high-quality speech, in the embodiment of the present application, after the Mel spectrum decoder, an adversarial generation training strategy is adopted, by using the discriminator to calculate the loss, and then updating the discriminator parameters and the parameters of the decoder to generate the Mel spectrum according to the calculation results. When the text enhancement features of the target text in the text fusion features are obtained by using the self-attention mechanism module, in S400, the text fusion features and the text enhancement features are input into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum.

[0092] Further, such as Figure 7 ,yes Figure 1 A flowchart of an embodiment after S400, after S400, the method further includes: S500, inputting the acquired target language Mel spectrum and the real target language Mel spectrum into the discriminator, and calculating the adversarial loss information of the adversarial network speech model; S600, using the adversarial loss information to update the parameters of the discriminator and the decoder.

[0093] In the embodiment of the present application, after the Mel spectrum decoder, an adversarial generation training strategy is adopted. The discriminator is used to calculate the loss, and then the discriminator parameters and the parameters of the decoder to generate the Mel spectrum are updated according to the calculation results, so that the generative adversarial network speech model can still generate high-quality target speech in the input text containing noise.

[0094] During training, a large amount of speech data in the target language is collected through various methods. It can be obtained from public speech databases, such as TIMI T, LibriSpeech, etc. These databases usually contain high-quality speech recordings in multiple languages. You can also record speech data yourself, such as inviting speakers whose native language is the target language to perform recording activities such as reading aloud and talking. Then, the collected speech samples are Fourier transformed and the Mel spectrum is calculated to obtain the Mel spectrum of the real target language. These Mel spectra can be used as training data to train models such as speech recognition and speech synthesis to improve the performance and accuracy of the model for the target language.

[0095] For example, suppose there is a speech synthesis task, the goal is to generate natural and fluent speech, and there is a discriminator to determine whether the generated speech is real. The generator of the generative adversarial network speech model makes slight perturbations to certain key audio features when generating speech waveforms. For example, for the fundamental frequency, formant and other features of the speech, random adjustments are made within a certain range.

[0096] For example, for a particular sentence, the generator may slightly increase or decrease its fundamental frequency, making the sound slightly different, but the overall semantics and expression remain the same. The generator can also try to combine multiple different speech styles or features to confuse the discriminator. It can combine the timbre characteristics, speech speed changes, emotional tendencies, etc. of different speakers for mixed generation.

[0097] Through these adversarial strategies, the generator constantly tries to break through the judgment of the discriminator, which can prompt the entire generative adversarial network speech model (GAN model) to gradually improve the quality and authenticity of the generated speech during the adversarial process.

[0098] In order to solve the above problems, Fig. 9 , Fig. 9According to the module structure diagram of a speech synthesis device 10 based on ByT5 of the present application; the embodiment of the present application also proposes a speech synthesis device 700 based on ByT5, and adopts a speech synthesis method based on ByT5, including:

[0099] The conversion module 701 is used to obtain a bit sequence of an input text and convert the bit sequence into a text feature vector using a ByT5 model;

[0100] The information embedding module 702 is used to obtain the part-of-speech information and syntactic structure information of the input text, perform information embedding encoding on the part-of-speech information and the syntactic structure information, and obtain a part-of-speech embedding vector and a syntactic structure embedding vector;

[0101] The feature fusion module 703 is used to sum the text feature vector, the part-of-speech embedding vector, and the syntactic structure embedding vector and input them into the fully connected layer of the generative adversarial network speech model for linear operation to obtain a text output vector, and input the text output vector into an activation function for feature fusion through nonlinear operation to obtain a text fusion feature;

[0102] The synthesis module 704 is used to input the text fusion features into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, input the target text MEL spectrum into the vocoder for speech synthesis, and obtain the target speech waveform.

[0103] By obtaining the bit sequence of the input text, the ByT5 model is used to convert the bit sequence into a text feature vector. The text feature vector, part-of-speech embedding vector, and syntactic structure embedding vector are summed and then input into the generative adversarial network speech model for feature fusion and decoding. This solves the problem of the token-based model being restricted by the dictionary, and can directly convert the original text into a unified bit representation in multiple languages, removing the complex and lengthy text preprocessing process. This can greatly reduce error accumulation and amortize the cost of operating directly on the original text. By integrating syntactic and structural information based on the powerful general processing capabilities of ByT5, it can effectively improve the quality, naturalness, and rhythm of the synthesized speech.

[0104] In order to solve the above problems, Fig.10 , Fig.10 A schematic diagram of a module structure of a computer device according to the present application. The present application also proposes a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of a speech synthesis method based on ByT5.

[0105] In order to solve the above problems, an embodiment of the present application further proposes a computer-readable storage medium, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor, the steps of a speech synthesis method based on ByT5 are implemented.

[0106] In order to solve the above problems, the embodiment of the present application also proposes a computer device, which adopts the following technical solution: it includes a memory, a communication module and a processor, the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of a speech synthesis method based on ByT5.

[0107] The computer device may be a computer, server, workstation or other device, or a mobile phone, tablet, vehicle-mounted mobile terminal or other device with program execution capability. The internal structure diagram of the computer device may be as follows: Fig.10 As shown, Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. The computer device includes a processor, a memory and a communication module. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage medium stores an operating system, instructions or codes. The internal memory provides an environment for the operation of the operating system and instructions or codes in the non-volatile storage medium. When the instructions or codes are executed by the processor, the functions or steps of a method for fine-tuning the above-mentioned pre-trained model are implemented. The communication module of the computer device may include a network interface and / or a wireless communication module, and the computer device may communicate with other devices or service platforms through the communication module. In addition, the computer device may also include a display screen and an input device, etc.

[0108] Among them, the memory is used to store a computer program, the computer program includes program instructions, the processor is configured to call the program instructions, and the processor implements the following steps when executing instructions or codes: S100, obtain a bit sequence of an input text, and use the ByT5 model to convert the bit sequence into a text feature vector; S200, obtain the part-of-speech information and syntactic structure information of the input text, perform information embedding encoding on the part-of-speech information and the syntactic structure information, and obtain a part-of-speech embedding vector and a syntactic structure embedding vector; S300, sum the text feature vector, the part-of-speech embedding vector, and the syntactic structure embedding vector, and input them into the fully connected layer of the generative adversarial network speech model for linear operation to obtain a text output vector, and input the text output vector into an activation function for feature fusion through nonlinear operation to obtain a text fusion feature; S400, input the text fusion feature into the decoder of the generative adversarial network speech model to obtain a target text MEL spectrum, and input the target text MEL spectrum into a vocoder for speech synthesis to obtain a target speech waveform.

[0109] In order to solve the above problems, the embodiment of the present application further proposes a computer-readable storage medium, which adopts the following technical solution:

[0110] The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor executes an adaptive parameter efficient fine-tuning method.

[0111] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, Figures 1 to 8 The speech synthesis method based on ByT5 provided in each step can be specifically referred to the implementation method provided in the above steps, which will not be repeated here.

[0112] The computer-readable storage medium may be a device for generating a video of a speaking person with a standardized lip shape provided by any of the aforementioned embodiments, or an internal storage unit of the terminal device, such as a hard disk or memory of a computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., equipped on the computer device.

[0113] Furthermore, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0114] However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0115] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0116] Compared with the existing technology, by obtaining the bit sequence of the input text, the ByT5 model is used to convert the bit sequence into a text feature vector, and the text feature vector, part-of-speech embedding vector, and syntactic structure embedding vector are summed and then input into the generative adversarial network speech model in sequence for feature fusion and decoding. This solves the problem that the token-based model is limited by the dictionary, and can directly convert the original text into a unified bit representation in multiple languages, removing the complex and lengthy text preprocessing process, which can greatly reduce error accumulation and amortize the cost of operating directly on the original text. It uses the powerful general processing capabilities of ByT5 to integrate syntactic and structural information, which can effectively improve the quality, naturalness and rhythm of the synthesized speech.

[0117] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0118] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A speech synthesis method based on ByT5, characterized in that it includes: Obtain a bit sequence of an input text, and convert the bit sequence into a text feature vector using a ByT5 model; Obtaining part-of-speech information and syntactic structure information of the input text, performing information embedding encoding on the part-of-speech information and syntactic structure information, and obtaining a part-of-speech embedding vector and a syntactic structure embedding vector; The text feature vector, the part-of-speech embedding vector, and the syntactic structure embedding vector are summed and input into the fully connected layer of the generative adversarial network speech model for linear operation to obtain a text output vector, and the text output vector is input into an activation function for feature fusion through nonlinear operation to obtain a text fusion feature; The text fusion feature is input into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, and the target text MEL spectrum is input into the vocoder for speech synthesis to obtain the target speech waveform.

2. The speech synthesis method based on ByT5 according to claim 1, characterized in that: After acquiring the text fusion feature, before inputting the text fusion feature into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, the method further includes: Inputting the text fusion features into the self-attention mechanism module of the generative adversarial network speech model; Using the self-attention mechanism module to obtain text enhancement features of the target text in the text fusion features, the text enhancement features include semantic features and position information of the text structure; The step of inputting the text fusion feature into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum includes: The text fusion feature and the text enhancement feature are input into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum.

3. The speech synthesis method based on ByT5 according to claim 2, characterized in that: After inputting the text fusion feature into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, inputting the target text MEL spectrum into the vocoder for speech synthesis to obtain the target speech waveform, the method further includes: Input the acquired Mel spectrum of the target language and the real Mel spectrum of the target language into the discriminator, and calculate the adversarial loss information of the generative adversarial network speech model; The adversarial loss information is used to update parameters of the discriminator and the decoder.

4. The speech synthesis method based on ByT5 according to claim 1, characterized in that: The step of obtaining a bit sequence of input text comprises: Obtaining input text, performing character encoding on the input text, and obtaining a digital encoding value corresponding to each character in the input text; Convert the digital code value corresponding to each character into a binary number; The binary numbers are connected in sequence to obtain a bit sequence of the input text.

5. The speech synthesis method based on ByT5 according to claim 4, characterized in that: The step of converting the bit sequence into a text feature vector by using the ByT5 model includes: Inputting the bit sequence of the input text into the input layer of the ByT5 model for preprocessing to obtain a standard text bit sequence; The standard text bit sequence is input into the embedding layer of the ByT5 model, and the standard text bit sequence is mapped into a text feature vector through the embedding layer.

6. The speech synthesis method based on ByT5 according to claim 1, characterized in that: The obtaining of part-of-speech information and syntactic structure information of the input text includes: Get the HanLP part-of-speech tagging tool; Using the HanLP part-of-speech tagging tool to create a part-of-speech tagger and a syntactic analyzer; The part-of-speech tagger is used to obtain the part-of-speech information of each word in the input text, and the syntactic analyzer is used to obtain the syntactic structure information of the input text.

7. The speech synthesis method based on ByT5 according to claim 6, characterized in that: The method of using the HanLP part-of-speech tagging tool to create a part-of-speech tagger and a syntactic analyzer includes: Import the classes corresponding to the part-of-speech tagger and syntactic analyzer into the HanLP part-of-speech tagging tool; The HanLP part-of-speech tagging tool calls the class corresponding to the part-of-speech tagger to create a part-of-speech tagger, and uses the part-of-speech tagger to output each word and its corresponding part of speech. The HanLP part-of-speech tagging tool calls the class corresponding to the syntactic analyzer to create a syntactic analyzer, and uses the syntactic analyzer to output a syntactic analysis result.

8. A speech synthesis device based on ByT5, using the speech synthesis method based on ByT5 according to any one of claims 1 to 7, characterized in that: include: A conversion module, used for obtaining a bit sequence of an input text, and converting the bit sequence into a text feature vector using a ByT5 model; An information embedding module, used to obtain the part-of-speech information and syntactic structure information of the input text, perform information embedding encoding on the part-of-speech information and syntactic structure information, and obtain a part-of-speech embedding vector and a syntactic structure embedding vector; A feature fusion module is used to sum the text feature vector, the part-of-speech embedding vector, and the syntactic structure embedding vector and input them into the fully connected layer of the generative adversarial network speech model for linear operation to obtain a text output vector, and input the text output vector into an activation function for feature fusion through nonlinear operation to obtain a text fusion feature; A synthesis module is used to input the text fusion feature into the decoder of the generative adversarial network speech model to obtain the target text MEL spectrum, input the target text MEL spectrum into the vocoder for speech synthesis, and obtain the target speech waveform.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the steps of the ByT5-based speech synthesis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech synthesis method based on ByT5 as described in any one of claims 1 to 7 are implemented.

Images