Method and device for synthesizing text into speech, computer equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a text speech synthesis method and device, computer equipment and a storage medium, which are applied to a financial transaction information speech synthesis scene and a medical diagnosis sheet information speech synthesis scene. Preprocessing the initial text audio pair to generate a training data set and a verification data set; carrying out optimization processing on an autoregression generation model based on the training data set by adopting a maximum likelihood estimation mode so as to carry out model training on the autoregression generation model and generate a target speech synthesis model; obtaining a to-be-synthesized text, and generating a mark sequence in an autoregression mode based on the to-be-synthesized text through the target speech synthesis model; and decoding the mark sequence to generate target voice information. The method is beneficial to improving the accuracy and efficiency of synthesizing the speech by the text.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular, to a method, device, computer device and storage medium for text-to-speech synthesis. Background Art

[0002] With the development of human-computer voice interaction technology, the application scope of text-to-speech synthesis is becoming wider and wider. The text-to-speech synthesis method can be applied to the fields of fintech and healthcare. For example, it can be applied to scenarios such as intelligent customer service, voice semantics, and intelligent robot responses, and can also be used in the scenario of medical order information voice synthesis. In the application scenario of voice-based medical information service for visually impaired patients and elderly patients, complex medical texts (such as inspection reports, medication guides, postoperative care instructions) are converted into natural voices to assist patients in obtaining information, enabling patients to accurately obtain key information such as medication dosages and follow-up appointment times through voice, reducing the risk of human misreading. Another example is in the application scenario of real-time voice notification of bank customer account changes. Transaction texts are converted into voices in real time and pushed through phone calls or APPs, enabling users to obtain high-risk transaction alerts in a timely manner through voice and enhancing the security of funds.

[0003] In text-to-speech synthesis methods, Transformer dominates in autoregressive generative modeling, but its self-attention mechanism may not be the best choice in voice (TTS) tasks. Research shows that Transformer tends to focus on local information or cross-attention between text and audio, which leads to waste of computing resources and insufficient model stability. In addition, the high complexity of self-attention and the high frame rate of audio codecs limit the training and inference efficiency of long contexts, especially when dealing with long texts, where it performs poorly. However, neural network encoder-decoder language models have achieved state-of-the-art performance in text-to-speech (TTS) synthesis tasks, thanks to scalable architectures such as autoregressive transformers and the application of large-scale voice datasets. By treating voice cloning as a prompt continuation task, these models perform well in cloning using short audio samples. However, this method has limitations when dealing with a large number of or longer voice segments, because the splicing of the source voice and the target voice must conform to the maximum context length determined during training, resulting in low efficiency of text-to-speech synthesis. Therefore, there is an urgent need for a text-to-speech synthesis method to improve the accuracy and efficiency of text-to-speech synthesis. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a method, device, computer device and storage medium for text-to-speech synthesis to improve the accuracy and efficiency of text-to-speech synthesis.

[0005] To solve the above technical problems, the embodiments of the present application provide a method for text-to-speech synthesis, including:

[0006] Obtain an initial text-audio pair, and preprocess the initial text-audio pair to generate a training data set and a validation data set;

[0007] Optimize an autoregressive generation model based on the training data set by using maximum likelihood estimation to perform model training on the autoregressive generation model to generate a target speech synthesis model;

[0008] Obtain a text to be synthesized, and generate a token sequence in an autoregressive manner based on the text to be synthesized through the target speech synthesis model;

[0009] Generate target speech information by decoding the token sequence.

[0010] To solve the above technical problems, an embodiment of the present application provides a text-to-speech device, including:

[0011] An initial data acquisition module, configured to obtain an initial text-audio pair, and preprocess the initial text-audio pair to generate a training data set and a validation data set;

[0012] A model training module, configured to optimize an autoregressive generation model based on the training data set by using maximum likelihood estimation to perform model training on the autoregressive generation model to generate a target speech synthesis model;

[0013] A token sequence generation module, configured to obtain a text to be synthesized, and generate a token sequence in an autoregressive manner based on the text to be synthesized through the target speech synthesis model;

[0014] A speech information generation module, configured to generate target speech information by decoding the token sequence.

[0015] To solve the above technical problems, a technical solution adopted by the present invention is: to provide a computer device, including one or more processors; a memory for storing one or more programs, so that one or more processors implement the text-to-speech method described in any one of the above.

[0016] To solve the above technical problems, a technical solution adopted by the present invention is: a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the text-to-speech method described in any one of the above is implemented.

[0017] An embodiment of the present invention provides a text-to-speech method, apparatus, computer device, and storage medium. The method includes: obtaining an initial text-audio pair, and preprocessing the initial text-audio pair to generate a training data set and a validation data set; optimizing an autoregressive generation model based on the training data set in a maximum likelihood estimation manner to perform model training on the autoregressive generation model to generate a target speech synthesis model; obtaining a text to be synthesized, and generating a tag sequence in an autoregressive manner based on the text to be synthesized through the target speech synthesis model; and generating target speech information by decoding the tag sequence. In the embodiment of the present invention, the autoregressive generation model is optimized based on the training data set in a maximum likelihood estimation manner to perform model training on the autoregressive generation model, so that the speech synthesis model can accurately synthesize the speech corresponding to the text; at the same time, by generating the tag sequence in an autoregressive manner for the text to be synthesized, the tag sequence can be generated quickly and accurately, which is beneficial to improving the accuracy and efficiency of text-to-speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the solutions in the present application, the following will briefly introduce the drawings required for the description of the embodiments of the present application. Obviously, the following drawings are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 is a schematic diagram of an application environment of the text-to-speech method according to an embodiment of the present invention;

[0020] Figure 2 is a flowchart of the implementation of the text-to-speech method provided by the embodiment of the present application;

[0021] Figure 3 is Figure 2 a schematic flowchart of a specific implementation manner of step S1 in

[0022] Figure 4 is Figure 2 a schematic flowchart of a specific implementation manner of step S2 in

[0023] Figure 5 is Figure 2 a schematic flowchart of a specific implementation manner of step S3 in

[0024] Figure 6 is Figure 5 a schematic flowchart of a specific implementation manner of step S32 in

[0025] Figure 7 is Figure 2Schematic flowchart of a specific implementation of step S4 in

[0026] Figure 8 Schematic diagram of a text-to-speech synthesis device provided by an embodiment of the present application;

[0027] Figure 9 Schematic diagram of a computer device provided by an embodiment of the present application. Specific implementation

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs; the terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second", etc. in the specification and claims of this application or the above drawings are used to distinguish different objects and not to describe a specific order.

[0029] Reference to "embodiment" herein means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of this application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0030] In order to enable those skilled in the art to better understand the solution of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0031] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0032] It should be noted that the text-to-speech synthesis method provided by the embodiments of this application is generally executed by a server. Correspondingly, the text-to-speech synthesis device is generally configured in the server.

[0033] The text-to-speech synthesis method provided by the embodiments of the present invention can be applied in, for example Figure 1In the application environment, the client communicates with the server through the network. The server can receive the text-to-speech synthesis request through the client, obtain the text to be synthesized; generate a token sequence in an autoregressive manner based on the text to be synthesized through a pre-constructed target speech synthesis model; generate the target speech information by decoding the token sequence. The server in the present invention can send the template speech information to the client. Among them, the client can be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, and portable wearable devices. The server can be implemented by an independent server or a server cluster composed of multiple servers. The present invention will be described in detail below through specific embodiments.

[0034] Please refer to Figure 2 , Figure 2 which shows a specific implementation of the text-to-speech synthesis method.

[0035] It should be noted that if there are substantially the same results, the method of the present invention is not limited to Figure 2 the process sequence shown, and the method includes the following steps:

[0036] S1: Obtain the initial text-audio pairs, and preprocess the initial text-audio pairs to generate a training data set and a validation data set.

[0037] Specifically, a large number of initial text-audio pairs are obtained, where the initial text-audio pairs include sample texts and the corresponding audio of the sample texts. Then, in order to improve the accuracy and stability of the subsequent speech synthesis model, it is necessary to preprocess the initial text-audio pairs, and then divide the preprocessed text-audio pairs to obtain a training data set and a validation data set.

[0038] Please refer to Figure 3 , Figure 3 which shows a specific implementation of step S1, described in detail as follows:

[0039] S11: Obtain the initial text-audio pairs.

[0040] S12: Respectively perform text preprocessing and audio preprocessing on the text-audio pairs to generate target text-audio pairs.

[0041] S13: Divide the target text-audio pairs according to a preset division ratio to generate the training data set and the validation data set.

[0042] Specifically, text preprocessing includes word segmentation, stop word removal, stemming / lemmatization, text cleaning and text vectorization. Word segmentation: Split the text into independent words or sub-word units so that the model can better understand and process them. Stop word removal: Remove meaningless words in the text, such as "的", "了", etc., to reduce data redundancy. Stemming / lemmatization (for languages ​​such as English): Restore words to their basic form, such as restoring "running" to "run", to improve the generalization ability of the model. Text cleaning: Remove special characters, extra spaces, HTML tags, etc. from the text to make the text neater. Text vectorization: Convert text into a vector representation so that the model can process it, which involves the use of word embedding technology such as Word2Vec, BERT, etc.

[0043] Audio preprocessing includes noise reduction, standardization, segmentation, feature extraction, and alignment. Noise reduction: remove background noise from the audio to improve the clarity of speech. Standardization: adjust the amplitude of the audio signal to an appropriate range to ensure that the model can process it stably. Segmentation: split long audio files into shorter segments so that the model can process them more efficiently. Feature extraction: extract useful features from the audio signal, such as Mel-frequency cepstral coefficients (MFCC), spectrograms, etc., so that the model can learn the speech characteristics of the audio. Alignment: For text-audio pairs, it is necessary to ensure that the text and audio are aligned in time, that is, each word or phrase in the text corresponds to the corresponding speech segment in the audio. Furthermore, in order to improve the generalization ability of the model, the text and audio data can be enhanced. For example, for text, new text can be generated by synonym replacement, sentence reorganization, etc.; for audio, new audio can be generated by changing the speaking speed, pitch, etc.

[0044] According to the preset division ratio, the target text audio pair is divided into data to generate the training data set and the verification data set. It should be noted that the division ratio is set according to the actual situation and is not limited here. In a specific embodiment, the target text audio pair is divided into a training data set and a verification data set according to a ratio of 9:1. In another embodiment, the target text audio pair is divided into a training data set, a verification data set and a test data set according to a ratio of 8:1:1, wherein the test data set is used to test the speech synthesis model.

[0045] S2: Optimizing the autoregressive generation model based on the training data set by using maximum likelihood estimation to perform model training on the autoregressive generation model and generate a target speech synthesis model.

[0046] In the embodiment of the present application, the speech synthesis model is an autoregressive generation model p θ, aiming to approximate the distribution of the neural audio codec token sequence (mark sequence) c, which is conditioned on the text input x. Specifically, the goal of the speech synthesis model is to learn the following conditional probability distribution: p θ (c|x). That is, the speech synthesis model p θ is the probability distribution of generating the audio codec token sequence c given the text input x.

[0047] Please refer to Figure 4 , Figure 4 which shows a specific implementation of step S2, described in detail as follows:

[0048] S21: Initialize the memory matrix of the gated linear attention in the autoregressive generation model.

[0049] In the embodiments of the present application, the gated linear attention (GLA) replaces the continuously expanding kv-cache in the traditional Transformer with a memory matrix S of a fixed size, achieving linear complexity. During the model training process, the states of all layers are initialized to zero, i.e., S t = 0. This type of memory can be domain-adapted or instruction-tuned for large language models through parameter-efficient fine-tuning (PEFT). Since the state encodes past information without the need to expand along the time axis, it provides a compact alternative for prompt tuning. 0

[0050] S22: Optimize the autoregressive generation model based on the training dataset by using maximum likelihood estimation to maximize the conditional probability of the text-audio pairs in the training dataset.

[0051] Specifically, during the model training process, maximum likelihood estimation (MLE) is used to optimize the model parameters θ, and the goal is to maximize the conditional probability of the text-audio pairs in the training dataset. Among them, the training objective of the autoregressive generation model is expressed as:

[0052]

[0053] where θ is the model parameter, is the training dataset, (x, c) is the text-audio pair, and c is the mark sequence.

[0054] In the embodiments of the present application, by maximizing the log probability logp θ (c|x), the model can better generate the audio codec token sequence x corresponding to the given text x.

[0055] S23: Calculate the gradient of the loss function using the backpropagation algorithm, and update the model parameters of the autoregressive generation model using an optimizer until the model loss value reaches a preset threshold or the number of iterations reaches a preset number, obtaining an initial speech synthesis model.

[0056] Specifically, after generating the audio codec token sequence x, decode the token sequence x to obtain the actual audio signal corresponding to the input training data text, calculate the loss function between this actual audio signal and the audio in the text-audio pair to obtain the model loss. Then calculate the gradient of the loss function using the backpropagation algorithm, and update the model parameters of the autoregressive generation model using an optimizer until the model loss value reaches a preset threshold or the number of iterations reaches a preset number, obtaining an initial speech synthesis model. It should be noted that the preset threshold and the preset number of iterations are set according to the actual situation and are not limited here.

[0057] S24: Validate the initial speech synthesis model using the validation dataset to generate the target speech synthesis model.

[0058] Specifically, after training the model, validate the initial speech synthesis model using the validation dataset to adjust the model parameters, thereby obtaining the target speech synthesis model.

[0059] Furthermore, in the inference stage of the target speech synthesis model, the model generates the audio codec token sequence (tag sequence) x one by one in an autoregressive manner. Specifically, given the text input and the partially generated token sequence c 1：t-1 , the model generates the probability of the next token c t sequence as:

[0060] p θ (c t | c 1：t-1 , x);

[0061] By continuously repeating this tag generation process, the model gradually generates a complete audio codec token sequence and converts it into an audible speech signal through a neural audio codec.

[0062] S3: Obtain the text to be synthesized, and generate a tag sequence based on the text to be synthesized in an autoregressive manner using the target speech synthesis model.

[0063] Specifically, the model architecture of the target speech synthesis model includes a text encoder, an acoustic model, and a decoder. The text encoder adopts a non-causal Transformer encoder structure, uses the self-attention mechanism as the temporal mixing operation, and applies SwiGLU as the feed-forward network. RoPE positional encoding is introduced in the encoder. Acoustic model: The acoustic model includes an audio encoder and an audio decoder, and adopts a causal Transformer architecture, where GLA (gated linear attention) is used as the temporal mixing operation, SwiGLU is used as the feed-forward network, and positional encoding is not used. Decoder: The decoder receives the input from the audio encoder and uses a cross-attention layer between the outputs of the text encoder and the audio encoder. To enhance robustness, position-aware cross-attention is adopted, and the sine positional encoding is replaced with convolutional positional encoding to improve training stability.

[0064] In the embodiment of the present application, the text to be synthesized is obtained, then the text encoder in the candy target synthesis model edits the text to be synthesized, then audio encoding is performed through the audio encoder, the audio encoding is decoded through the audio decoder to generate a token sequence, and finally the token sequence is decoded through the decoder to generate the target speech information.

[0065] Further, in the model inference process, the embodiment of the present application uses the top-k sampling method, the k value is set to 100, and the EOS (end of sentence) token is processed as an additional token of the audio codebook.

[0066] The embodiment of the present application is robust to the selection of hyperparameters for different data sets. In practice, the same learning rate λ = 0.1 is used for all examples, and two passes are made over the data set, with a batch size of 8 sentences and a maximum number of steps of 40 steps. When restricted to 3 - 15 minutes of speech data, the state matrix can be parameterized as a rank-1 matrix (i.e., ), which reduces the parameter set to two vectors per head and per layer without significantly affecting performance.

[0067] Please refer to Figure 5 , Figure 5 which shows a specific implementation manner of step S3, described in detail as follows:

[0068] S31: Obtain the text to be synthesized, perform text encoding on the text to be synthesized through the text encoder, and use RoPE positional encoding to retain the order information in the text to be synthesized to generate the target text encoding, where the text encoder adopts a non-causal Transformer structure, uses the self-attention mechanism to process the text to be synthesized, and applies SwiGLU as the feed-forward network.

[0069] S32: Based on the target text encoding, generate the token sequence in an autoregressive manner through the target speech synthesis model.

[0070] Specifically, the purpose of the text encoder is to convert text into a numerical form that can be processed by subsequent models. RoPE Position Encoding: RoPE (Rotary Position Embedding) is a position encoding method used to preserve the order information of text in text encoding. In natural language, the order of words is very important, and RoPE can help the model understand this. Generation of Target Text Encoding: Combine text encoding and RoPE position encoding to generate target text encoding containing text content and order information. Non-Causal Transformer Structure: The text encoder adopts a non-causal Transformer structure. This means that during the encoding process, the model can consider the context information before and after the current word simultaneously. Self-Attention Mechanism: The core of the Transformer structure is the self-attention mechanism, which allows the model to pay attention to all other words in the input sequence when processing each word. SwiGLU as the Feed-Forward Network: SwiGLU (Swish-gated Linear Unit) is a special feed-forward network structure used in the feed-forward layer of the Transformer. It combines the Swish activation function and the gating mechanism, which helps improve the expressive power of the model.

[0071] Please refer to Figure 6 , Figure 6 which shows a specific implementation manner of step S32, described in detail as follows:

[0072] S321: Through the audio encoder in the target speech synthesis model, perform speech encoding on the target text encoding to generate target audio encoding.

[0073] S322: Based on the target audio encoding, generate an initial token sequence through the audio decoder in the target speech synthesis model.

[0074] S323: Generate the next token sequence in an autoregressive manner based on the initial token sequence until all sequences are generated to obtain the token sequence.

[0075] Specifically, both the audio encoder and the audio decoder adopt a causal Transformer architecture, where GLA (Gated Linear Attention) is used as the temporal mixing operation, SwiGLU is used as the feed-forward network, and position encoding is not used. In the embodiments of the present application, based on the target audio encoding, an initial token sequence is generated through the audio decoder in the target speech synthesis model. The next token sequence is generated in an autoregressive manner based on the initial token sequence until all sequences are generated to obtain the token sequence.

[0076] S4: Generate target speech information by decoding the marked sequence.

[0077] Specifically, the decoder receives the input from the audio encoder, that is, receives the marked sequence, and uses a cross-attention layer between the outputs of the text encoder and the audio encoder. The marked sequence is subjected to attention calculation through the cross-attention layer, and then feature decoding is performed to generate target speech information. To enhance robustness, position-aware cross-attention is adopted, and the sine position encoding is replaced by convolutional position encoding.

[0078] Please refer to Figure 7 , Figure 7 which shows a specific implementation manner of step S4, described in detail as follows:

[0079] S41: Perform attention calculation on the marked sequence through the cross-attention layer to generate target features.

[0080] S42: Perform feature decoding on the target features by the decoder based on convolutional position encoding to generate the target speech information.

[0081] Specifically, to enhance robustness, position-aware cross-attention is adopted, and the sine position encoding is replaced by convolutional position encoding. Therefore, in the embodiments of the present application, attention calculation is performed on the marked sequence through the cross-attention layer to generate target features. Feature decoding is performed on the target features by the decoder based on convolutional position encoding to generate the target speech information.

[0082] In a specific embodiment, the embodiments of the present application can be applied to medical scenarios. For example, in the application scenario of voice conversion of medical information for visually impaired patients and elderly patients, patients often face the problem of being unable to intuitively read paper reports or electronic medical records. The embodiments of the present application can be applied to convert complex medical texts (such as inspection reports, medication guides, postoperative care instructions) into natural speech to assist patients in obtaining information. For example, a diabetic patient needs to receive daily insulin injection dose reminders, and the system converts the doctor's order text into understandable voice instructions. In this scenario, standardized doctor's order texts from the hospital's endocrinology department (such as "subcutaneous injection of 6 units of insulin before breakfast every day") and corresponding professional recordings are collected, covering key information such as dose, drug name, and time. During preprocessing, a medical dictionary is used to align the text with the audio, background noise (such as the interference sound of an electrocardiogram machine) is filtered, and the pronunciation rules of rare terms (such as "insulin glargine") are marked. Then, the above data is used to train the target speech synthesis model for this medical scenario. During actual operation, the text to be synthesized is input. For example, "The dressing needs to be changed three days after the operation. Please return for a follow-up visit if the body temperature exceeds 38.5°C". The target speech synthesis model generates a sequence token by token in an autoregressive manner, and captures the relevance between "body temperature" and "follow-up visit" through the attention mechanism. During the decoding stage, a medical-specific vocoder is used to emphasize the decimal point in "38.5°C" with stress and generate the voice information corresponding to the text to be synthesized, so that patients can accurately obtain key information such as medication dose and follow-up visit time through voice, reducing the risk of human misreading.

[0083] In another specific embodiment, the embodiments of the present application can be applied to financial scenarios. For example, in the application scenario of real-time voice notifications for bank customer account changes, customers need to know the account dynamics in a timely manner but may not be able to view text information at any time. The embodiments of the present application can convert transaction texts (such as "Your account with the last four digits 8899 transferred and spent xxx yuan at 14:05") into voice in real time and push them through phone calls or APPs. For example, when an investment product matures, the system automatically generates a voice reminder for the income to be credited to the account. In this scenario, bank transaction record texts (including account tail numbers, amounts, and times) and multi-scenario recordings by recorders (such as a rapid tone to simulate a fraud alert and a steady tone for balance notifications) are collected. During the preprocessing stage, the digital expressions are standardized, and "$1,234.56" is uniformly converted to "one thousand two hundred thirty-four point five six US dollars", and the stress positions are marked to prevent ambiguity. Then, the above data is used to train the target speech synthesis model for this financial scenario, so as to meet the requirement of text-to-speech synthesis in this scenario. Customers can obtain high-risk transaction alerts in a timely manner through voice in scenarios such as driving, improving the security of funds.

[0084] In the embodiments of the present application, an initial text-audio pair is obtained, and the initial text-audio pair is preprocessed to generate a training data set and a validation data set; the autoregressive generation model is optimized based on the training data set by means of maximum likelihood estimation to train the autoregressive generation model and generate a target speech synthesis model; a text to be synthesized is obtained, and a token sequence is generated autoregressively based on the text to be synthesized through the target speech synthesis model; and the target speech information is generated by decoding the token sequence. In the embodiments of the present invention, the autoregressive generation model is optimized based on the training data set by means of maximum likelihood estimation to train the autoregressive generation model, so that the speech synthesis model can accurately synthesize the speech corresponding to the text; at the same time, by generating the token sequence autoregressively for the text to be synthesized, the token sequence can be generated quickly and accurately, which is beneficial to improving the accuracy and efficiency of text-to-speech synthesis.

[0085] The embodiments of the present application combine the gated linear attention mechanism and initial state tuning, demonstrating performance comparable to or even better than that of powerful neural encoder-decoder language model baselines. By combining gated linear attention and initial state tuning, efficient speech synthesis and speech cloning are achieved. Its advantages in zero-shot learning and parameter-efficient fine-tuning make it highly competitive in the field of speech synthesis.

[0086] Please refer to Figure 8 , as an implementation of the above Figure 2 shown method, an embodiment of a text-to-speech device is provided in the present application. This device embodiment corresponds to the Figure 2 shown method embodiment, and this device can be specifically applied to various electronic devices.

[0087] As Figure 8 shown, the text-to-speech device of this embodiment includes: an initial data acquisition module 51, a model training module 52, a token sequence generation module 53, and a speech information generation module 54, where:

[0088] The initial data acquisition module 51 is configured to obtain an initial text-audio pair and preprocess the initial text-audio pair to generate a training data set and a validation data set;

[0089] The model training module 52 is configured to optimize the autoregressive generation model based on the training data set by means of maximum likelihood estimation to train the autoregressive generation model and generate a target speech synthesis model;

[0090] The token sequence generation module 53 is configured to obtain a text to be synthesized and generate a token sequence autoregressively based on the text to be synthesized through the target speech synthesis model;

[0091] A voice information generation module 54, configured to generate target voice information by decoding the marked sequence.

[0092] Further, the model training module 52 includes:

[0093] An initialization unit, configured to initialize the memory matrix of the gated linear attention in the autoregressive generation model;

[0094] An optimization unit, configured to optimize the autoregressive generation model based on the training data set by using maximum likelihood estimation to maximize the conditional probability of the text-audio pair in the training data set;

[0095] A backpropagation unit, configured to calculate the gradient of the loss function by using the backpropagation algorithm and update the model parameters of the autoregressive generation model by using an optimizer until the model loss value reaches a preset threshold or the number of iterations reaches a preset number, to obtain an initial voice synthesis model;

[0096] A verification unit, configured to verify the initial voice synthesis model through the verification data set to generate the target voice synthesis model.

[0097] Further, the training objective of the autoregressive generation model is expressed as:

[0098]

[0099] where θ is the model parameter, is the training data set, (x, c) is a text-audio pair, and c is a marked sequence.

[0100] Further, the marked sequence generation module 53 includes:

[0101] A text encoding unit, configured to obtain the text to be synthesized, perform text encoding on the text to be synthesized through a text encoder, and retain the order information in the text to be synthesized by using RoPE positional encoding to generate a target text encoding, where the text encoder adopts a non-causal Transformer structure, processes the text to be synthesized by using a self-attention mechanism, and applies SwiGLU as a feed-forward network;

[0102] A sequence generation unit, configured to generate the marked sequence in an autoregressive manner based on the target text encoding through the target voice synthesis model.

[0103] Further, the sequence generation unit includes:

[0104] An audio encoding unit, configured to perform voice encoding on the target text encoding through an audio encoder in the target voice synthesis model to generate a target audio encoding;

[0105] An initial token sequence generation unit for generating an initial token sequence based on the target audio encoding through audio decoding in the target speech synthesis model;

[0106] A token sequence completion unit for generating the next token sequence in an autoregressive manner based on the initial token sequence until all sequences are generated to obtain the token sequence.

[0107] Further, the speech information generation module 54 includes:

[0108] A target feature generation unit for performing attention calculation on the token sequence through a cross-attention layer to generate a target feature;

[0109] A feature decoding unit for performing feature decoding on the target feature through a decoder based on convolutional positional encoding to generate the target speech information.

[0110] Further, the initial data acquisition module 51 includes:

[0111] A text-audio pair acquisition unit for acquiring the initial text-audio pair;

[0112] A preprocessing unit for respectively performing text preprocessing and audio preprocessing on the text-audio pair to generate a target text-audio pair;

[0113] A data partitioning unit for partitioning the target text-audio pair according to a preset partitioning ratio to generate the training data set and the validation data set.

[0114] In the embodiment of the present application, an initial text-audio pair is acquired, and the initial text-audio pair is preprocessed to generate a training data set and a validation data set; the autoregressive generation model is optimized based on the training data set in a maximum likelihood estimation manner to perform model training on the autoregressive generation model to generate a target speech synthesis model; a text to be synthesized is acquired, and a token sequence is generated in an autoregressive manner based on the text to be synthesized through the target speech synthesis model; and the target speech information is generated by decoding the token sequence. The embodiment of the present invention optimizes the autoregressive generation model based on the training data set in a maximum likelihood estimation manner to perform model training on the autoregressive generation model, which can enable the speech synthesis model to accurately synthesize the speech corresponding to the text; at the same time, by generating the token sequence in an autoregressive manner for the text to be synthesized, the token sequence can be generated quickly and accurately, which is beneficial to improving the accuracy and efficiency of text-to-speech synthesis.

[0115] To solve the above technical problems, the embodiment of the present application also provides a computer device. For details, please refer to Figure 9 ,Figure 9 This is the basic structural block diagram of the computer device in this embodiment.

[0116] The computer device 7 includes a memory 71, a processor 72, and a network interface 73 that are communicatively connected to each other via a system bus. It should be noted that Figure 9 only the computer device 7 with three components, namely the memory 71, the processor 72, and the network interface 73, is shown. However, it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented. Among them, those skilled in the art of this technology can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0117] The computer device can be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The computer device can perform human-computer interaction with the user through a keyboard, a mouse, a remote control, a touchpad, a voice control device, or other means.

[0118] The memory 71 includes at least one type of readable storage medium. The readable storage medium includes flash memory, a hard disk, a multimedia card, a card-type memory (such as an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 71 can be an internal storage unit of the computer device 7, such as the hard disk or memory of the computer device 7. In other embodiments, the memory 71 can also be an external storage device of the computer device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 7. Of course, the memory 71 can also include both the internal storage unit and the external storage device of the computer device 7. In this embodiment, the memory 71 is generally used to store the operating system and various application software installed on the computer device 7, such as the program code of the text-to-speech synthesis method. In addition, the memory 71 can also be used to temporarily store various types of data that have been output or will be output.

[0119] The processor 72 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 72 is generally used to control the overall operation of the computer device 7. In this embodiment, the processor 72 is used to run the program code stored in the memory 71 or process data, such as running the program code of the above text-to-speech method to implement various embodiments of the text-to-speech method.

[0120] The network interface 73 may include a wireless network interface or a wired network interface, and this network interface 73 is generally used to establish a communication connection between the computer device 7 and other electronic devices.

[0121] The present application also provides another implementation manner, that is, to provide a computer-readable storage medium storing a computer program, and the computer program can be executed by at least one processor to enable the at least one processor to execute the steps of a text-to-speech method as described above.

[0122] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-described embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation manner. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of the various embodiments of the present application.

[0123] Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The accompanying drawings show the preferred embodiments of the present application, but do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure content of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements for some of the technical features. Any equivalent structure directly or indirectly using the content of the specification and drawings of the present application in other related technical fields is equally within the scope of the patent protection of the present application.

Claims

1. A text-to-speech method, characterized in that: include: Acquire an initial text-audio pair, and preprocess the initial text-audio pair to generate a training data set and a verification data set; The autoregressive generation model is optimized based on the training data set by using a maximum likelihood estimation method to perform model training on the autoregressive generation model to generate a target speech synthesis model; Acquire the text to be synthesized, and generate a tag sequence in an autoregressive manner based on the text to be synthesized by the target speech synthesis model; The target speech information is generated by decoding the tag sequence.

2. The text-to-speech method according to claim 1, characterized in that: The method of optimizing the autoregressive generation model based on the training data set by adopting the maximum likelihood estimation method to perform model training on the autoregressive generation model and generate a target speech synthesis model includes: Initialize the memory matrix of the gated linear attention in the autoregressive generative model; The autoregressive generative model is optimized based on the training data set by using maximum likelihood estimation to maximize the conditional probability of the text-audio pairs in the training data set; The gradient of the loss function is calculated using a back-propagation algorithm, and the model parameters of the autoregressive generation model are updated using an optimizer until the model loss value reaches a preset threshold or the number of iterations reaches a preset number, thereby obtaining an initial speech synthesis model; The initial speech synthesis model is verified using the verification data set to generate the target speech synthesis model.

3. The text-to-speech method according to claim 2, characterized in that: The training objective of the autoregressive generative model is expressed as: Where θ is the model parameter, is the training data set, (x, c) is a text-audio pair, and c is a labeled sequence.

4. The text-to-speech method according to claim 1, characterized in that: The step of obtaining the text to be synthesized and generating a tag sequence in an autoregressive manner based on the text to be synthesized by using the target speech synthesis model includes: Obtain the text to be synthesized, perform text encoding on the text to be synthesized through a text encoder, and use RoPE position encoding to retain the order information in the text to be synthesized to generate a target text encoding, wherein the text encoder adopts a non-causal Transformer structure, uses a self-attention mechanism to process the text to be synthesized, and applies SwiGLU as a feedforward network; The tag sequence is generated in an autoregressive manner based on the target text encoding by the target speech synthesis model.

5. The text-to-speech method according to claim 4, characterized in that: The step of generating the tag sequence in an autoregressive manner based on the target text encoding by the target speech synthesis model comprises: Performing speech encoding on the target text code by using an audio encoder in the target speech synthesis model to generate a target audio code; Generating an initial tag sequence based on the target audio encoding by audio decoding in the target speech synthesis model; The next label sequence is generated based on the initial label sequence in an autoregressive manner until all sequences are generated to obtain the label sequence.

6. The text-to-speech method according to any one of claims 1 to 5, characterized in that: The step of generating target speech information by decoding the tag sequence comprises: Performing attention calculation on the tag sequence through a cross-attention layer to generate target features; The target feature is feature decoded by a decoder based on convolutional position coding to generate the target speech information.

7. The text-to-speech method according to any one of claims 1 to 5, characterized in that: The obtaining of the initial text-audio pair and preprocessing the initial text-audio pair to generate a training data set and a verification data set include: Obtaining the initial text-audio pair; Performing text preprocessing and audio preprocessing on the text-audio pair respectively to generate a target text-audio pair; The target text audio pair is divided into data according to a preset division ratio to generate the training data set and the verification data set.

8. A text-to-speech device, characterized in that: include: An initial data acquisition module is used to acquire initial text-audio pairs and pre-process the initial text-audio pairs to generate a training data set and a verification data set; A model training module, used for optimizing the autoregressive generation model based on the training data set by adopting a maximum likelihood estimation method, so as to perform model training on the autoregressive generation model and generate a target speech synthesis model; A tag sequence generation module, used for obtaining a text to be synthesized, and generating a tag sequence in an autoregressive manner based on the text to be synthesized by using the target speech synthesis model; The speech information generation module is used to generate target speech information by decoding the tag sequence.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the text-to-speech method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the text-to-speech method according to any one of claims 1 to 7 is implemented.

Images