Text-to-speech method, apparatus, device and storage medium

Abstract

Embodiments of the present application provide a text-to-speech method, device, equipment and storage medium. The method comprises: obtaining a target text and determining semantic information corresponding to the target text; processing the semantic information through a speech tokenizer in a target large language model to obtain discrete speech tokens corresponding to the target text, wherein the discrete speech tokens include encoding information corresponding to the semantic information; determining position information corresponding to the discrete speech tokens based on the encoding information; and predicting the discrete speech tokens through a vocoder in the target large language model to obtain speech signals corresponding to the discrete speech tokens; and sorting the speech signals according to the position information to obtain a target speech stream corresponding to the target text. Embodiments of the present application aim to achieve strong semantic association in the text-to-speech process and improve the accuracy and efficiency of text-to-speech.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method for text-to-speech conversion, a device for text-to-speech conversion, a computer device, and a computer-readable storage medium. Background Technology

[0002] Speech synthesis refers to the process of converting text into spoken audio. Currently, speech synthesis models are commonly used to perform this operation. Among related technologies, neural network-based speech synthesis techniques are the primary means of text-to-speech conversion.

[0003] Neural network-based speech synthesis systems are divided into two main categories: autoregressive and non-autoregressive models. Autoregressive models suffer from long synthesized audio time and poor quality, while non-autoregressive models are limited by inconsistent input and output lengths, making it difficult to align synthesized speech with the text. Therefore, they cannot achieve strong speech association, resulting in poor quality and low accuracy. Summary of the Invention

[0004] This application provides a method, apparatus, computer device, and computer-readable storage medium for text-to-speech conversion, aiming to achieve stronger semantic association and improve the accuracy and efficiency of text-to-speech conversion during the text-to-speech process.

[0005] To achieve the above objectives, this application also provides a method for text-to-speech conversion, comprising:

[0006] Obtain the target text and determine the semantic information corresponding to the target text;

[0007] The semantic information is processed by the speech segmenter in the target large language model to obtain the discrete speech token corresponding to the target text, wherein the discrete speech token includes the encoded information corresponding to the semantic information;

[0008] Based on the encoded information, the location information corresponding to the discrete speech token is determined; and the speech signal corresponding to the discrete speech token is obtained by predicting the discrete speech token through the vocoder in the target large language model.

[0009] Used to sort the speech signal according to the location information to obtain the target speech stream corresponding to the target text.

[0010] To achieve the above objectives, this application also provides a text-to-speech apparatus, comprising:

[0011] The acquisition module is used to acquire the target text and determine the semantic information corresponding to the target text;

[0012] The determination module is used to process the semantic information through the speech segmenter in the target large language model to obtain the discrete speech token corresponding to the target text, wherein the discrete speech token includes the encoded information corresponding to the semantic information;

[0013] The determining module is used to determine the location information corresponding to the discrete speech token based on the encoded information; and to predict the discrete speech token by using the vocoder in the target large language model to obtain the speech signal corresponding to the discrete speech token.

[0014] The conversion module is used to sort the speech signal according to the location information to obtain the target speech stream corresponding to the target text.

[0015] In addition, to achieve the above objectives, this application also provides a computer device, the computer device including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and, when executing the computer program, implement the steps of the text-to-speech method provided in any of the embodiments of this application.

[0016] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to implement the steps of the text-to-speech method provided in any of the embodiments of this application.

[0017] This application discloses a method, apparatus, computer device, and computer-readable storage medium for text-to-speech conversion. The method includes acquiring target text and determining the semantic information corresponding to the target text. Further, the semantic information can be processed by a speech segmenter in a target large language model to obtain discrete speech tokens corresponding to the target text, wherein the discrete speech tokens include encoded information corresponding to the semantic information. Based on the encoded information, the positional information corresponding to the discrete speech tokens can be determined; and the discrete speech tokens can be predicted by a vocoder in the target large language model to obtain the speech signal corresponding to the discrete speech tokens. Finally, the speech signals can be sorted according to the positional information to obtain the target speech stream corresponding to the target text. This application, by introducing a speech segmenter and a vocoder in a target large language model, can effectively process the semantic information of the target text and generate corresponding discrete speech tokens, ensuring accurate encoding and decoding of the semantic information. Simultaneously, by determining the positional information of the discrete speech tokens based on the encoded information and combining it with the prediction of the vocoder to generate high-quality speech signals, and further sorting them to obtain a coherent and natural speech stream, the accuracy and fluency of text-to-speech conversion are improved. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating a method for text-to-speech provided in an embodiment of this application;

[0020] Figure 2 This is a flowchart illustrating a text-to-speech method provided in an embodiment of this application;

[0021] Figure 3 This is a schematic block diagram of a text-to-speech device provided in an embodiment of this application;

[0022] Figure 4 This is a schematic block diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0023] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation. Furthermore, although functional modules are divided in the device diagram, in some cases, a different module division may be used.

[0025] The term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items, as well as all possible combinations, and includes such combinations.

[0026] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0027] like Figure 1 As shown, the text-to-speech method provided in this application embodiment can be applied to, for example... Figure 1The application environment shown includes a terminal device 110 and a server 120. The terminal device 110 can communicate with the server 120 via a network. Specifically, the server 120 can acquire target text and determine the semantic information corresponding to the target text; process the semantic information using a speech segmenter in the target large language model to obtain a discrete speech token corresponding to the target text, wherein the discrete speech token includes encoding information corresponding to the semantic information; determine the location information corresponding to the discrete speech token based on the encoding information; and predict the discrete speech token using a vocoder in the target large language model to obtain the speech signal corresponding to the discrete speech token; sort the speech signal according to the location information to obtain the target speech stream corresponding to the target text, thereby sending the target speech stream to the terminal device 110. The server 120 can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal device 110 may be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be connected directly or indirectly through wired or wireless communication, and this application does not impose any restrictions.

[0028] Please see Figure 2 , Figure 2 This is a flowchart illustrating a text-to-speech method provided in an embodiment of this application. Figure 2 As shown, the method of text-to-speech conversion can be achieved through steps S11 to S14.

[0029] Step S11: Obtain the target text and determine the semantic information corresponding to the target text.

[0030] For example, target text refers to input text that needs to be converted into speech. It can include any text data entered by the user, such as sentences, paragraphs, or articles. The text may include natural language descriptions, commands, or other formatted data.

[0031] For example, semantic information refers to the implicit meaning or content within a target text; it is deeper information extracted from the literal content of the text. Semantic information not only focuses on the surface meaning of words but also includes the text's context, sentence structure, emotional tone, and intent.

[0032] For example, natural language processing techniques, such as word embedding models and sentence vector models, can be used to convert target text into semantic vectors and capture its meaning. Extracting semantic information requires not only analyzing the meaning of individual words but also grammatical and syntactic analysis to understand the logic and context of the text's expression.

[0033] The above implementation method obtains the input target text and extracts the semantic information of the text through natural language processing technology, providing an accurate semantic basis for subsequent speech generation. This ensures that the generated speech not only contains correct word pronunciation but also accurately conveys the meaning and emotion of the text.

[0034] Step S12: Process the semantic information using the speech segmenter in the target large language model to obtain the discrete speech tokens corresponding to the target text.

[0035] Among them, discrete voice tokens include encoded information corresponding to semantic information.

[0036] It should be noted that discrete speech tokens are the result of a speech segmentation process. Discrete speech tokens are used to represent the discrete representation of semantic information in the speech synthesis process, similar to basic elements such as phonemes, tones, and rhythms in speech.

[0037] For example, discrete speech tokens also carry encoded information extracted from semantic information. This encoded information represents key features of the original semantic information, including: emotional expressions such as happiness, anger, and confusion; speech features such as pitch, speech rate, and volume to ensure the synthesized speech is fluent and natural; and contextual understanding to ensure coherent speech generation between consecutive sentences.

[0038] For example, a speech segmenter can generate discrete speech tokens suitable for speech synthesis by segmenting and encoding semantic information. These discrete speech tokens are then used to predict the actual speech signal, ultimately synthesizing it into a natural and fluent speech output.

[0039] The above implementation method can use a speech segmenter to process complex semantic information into discrete speech tokens. The discrete speech tokens not only represent the pronunciation units in the semantic information, but also carry encoded information such as emotion, intonation, and speech rate, which provides a basis for the generation of subsequent speech signals and ensures that the generated speech is not only accurate in content, but also conveys the emotions and tone contained in the text.

[0040] Step S13: Based on the encoding information, determine the location information corresponding to the discrete speech token; and predict the discrete speech token by using the vocoder in the target large language model to obtain the speech signal corresponding to the discrete speech token.

[0041] For example, discrete speech tokens have been processed by a speech segmenter, carrying corresponding encoded information. The encoded information not only represents the content of the speech segments (such as phonemes, emotion, pitch, etc.), but also marks the relative positions of each speech segment to ensure that the generated speech maintains the same order and intonation as the original text.

[0042] For example, location information refers to the order and position of each discrete speech token within the target speech stream. This ensures that the synthesized speech signal is coherent and conforms to the natural pronunciation order of the target text. By encoding this information, the precise position of each discrete speech token within the speech sequence can be extracted.

[0043] It's important to note that the vocoder is a key module in generating the final speech signal. The vocoder converts discrete speech tokens into a continuous speech waveform signal. By predicting the input discrete speech tokens, the vocoder generates a high-quality, natural speech signal.

[0044] For example, a vocoder processes discrete speech tokens sequentially according to their location information, predicting the corresponding speech waveform signals. The speech signal for each discrete speech token is generated step-by-step, forming a continuous speech stream. During the prediction process, the vocoder processes details of the speech, such as smoothing the speech waveform, adjusting pitch, and controlling speech rate, to ensure the generated speech is natural and fluent. Through the vocoder's prediction, each discrete speech token generates a corresponding speech signal segment. These signal segments are then combined according to the location information to form a complete target speech stream.

[0045] It should be understood that by predicting all discrete speech tokens and sorting them by location information, the final generated speech stream corresponds to the semantic information of the target text and can accurately convey the content and emotion of the original text. This speech stream can be directly used in speech synthesis applications, such as voice assistants, navigation systems, and audio content generation.

[0046] Step S14: Sort the speech signals according to the location information to obtain the target speech stream corresponding to the target text.

[0047] For example, based on the location information, all speech segments are ordered in the order they should appear. For instance, the segment with location information 1 should appear first, the segment with location information 2 should follow, and so on, until the last speech segment.

[0048] For example, in addition to sequence, positional information can also help determine the temporal intervals between individual speech segments. By adjusting the time length between each segment, it's possible to ensure a natural rhythm and maintain an appropriate speaking speed.

[0049] For example, after the speech segments are sorted, they are sequentially spliced ​​into a continuous speech stream. During splicing, transitions between speech segments are handled to ensure natural flow and avoid abrupt or disjointed situations.

[0050] The above implementation sorts and concatenates the discrete speech signal segments generated by the vocoder according to their location information, ultimately obtaining a complete and coherent target speech stream. Through precise control of the location information, the generated speech naturally reflects the text content and semantics, while maintaining the accuracy of rhythm, speed, and emotional expression. This step ensures that the speech synthesis output is of high quality and can meet various practical application requirements.

[0051] The text-to-speech method disclosed in this application includes acquiring target text and determining the semantic information corresponding to the target text. Further, the semantic information can be processed by a speech segmenter in the target large language model to obtain discrete speech tokens corresponding to the target text, wherein the discrete speech tokens include encoded information corresponding to the semantic information. Therefore, the positional information corresponding to the discrete speech tokens can be determined based on the encoded information; and the speech signal corresponding to the discrete speech tokens can be obtained by predicting the discrete speech tokens using a vocoder in the target large language model. Finally, the speech signals can be sorted according to the positional information to obtain the target speech stream corresponding to the target text. This application, by introducing a speech segmenter and a vocoder in the target large language model, can effectively process the semantic information of the target text and generate corresponding discrete speech tokens, ensuring accurate encoding and decoding of the semantic information. Simultaneously, by determining the positional information of the discrete speech tokens based on the encoded information and combining it with the prediction of the vocoder to generate high-quality speech signals, and further sorting them to obtain a coherent and natural speech stream, the accuracy and fluency of text-to-speech conversion are improved.

[0052] Optionally, determining the semantic information corresponding to the target text includes: performing preprocessing operations on the target text to obtain preprocessed target text, wherein the preprocessing operations include word segmentation, normalization, and stop word removal; analyzing the text structure information of the preprocessed target text, and determining the semantic information corresponding to the target text based on the text structure information.

[0053] For example, the target text can first be preprocessed to extract semantic information more accurately. Preprocessing operations include breaking the text down into lexical units (word segmentation), standardizing the text format (normalization), and removing stop words that do not significantly contribute to semantics. Furthermore, by analyzing the structural information of the preprocessed text (such as sentence structure and relationships between words), a more comprehensive understanding of the text's semantics can be achieved, allowing for the extraction of corresponding semantic information and providing an accurate foundation for subsequent speech generation.

[0054] The above embodiments, through text preprocessing and structural analysis, ensure more accurate and complete extraction of semantic information, providing high-quality semantic input for subsequent speech synthesis.

[0055] Alternatively, the discrete voice token is obtained using the following formula:

[0056]

[0057] Where, μ l For discrete voice tokens; ||·| |2 The L2 norm is used; VQ is the vector quantizer; h l is for hidden representation; C is for default codebook.

[0058] Optionally, the speech segmenter includes a vector quantizer and a convolutional encoder. The speech segmenter in the target large language model processes the semantic information to obtain the discrete speech token corresponding to the target text. This includes: obtaining a preset codebook and determining the target hidden representation corresponding to the semantic information through the convolutional encoder; determining the code vector with the smallest distance from the code vectors in the preset codebook to the target hidden representation through the vector quantizer, and determining the index of the code vector with the smallest distance as the discrete speech token corresponding to the target text.

[0059] It's important to note that a codebook is a collection of code vectors, each representing a different speech or semantic feature in the feature space. It can be understood as a discretized reference table, where each code vector represents a specific speech feature. The codebook provides a standard reference for converting the semantic information of text into discrete tokens, thereby mapping continuous semantic information to a discrete, quantifiable form.

[0060] Furthermore, the task of the convolutional encoder is to encode the semantic information of the input, transforming it into a target hidden representation suitable for subsequent quantization processing. The convolutional encoder can extract local features of the semantic information, such as relationships between words and contextual information. The output of the convolutional encoder is the target hidden representation, a continuous numerical vector that retains the core features of the semantic information. These feature vectors are not yet discretized and require further processing.

[0061] Furthermore, the task of the vector quantizer is to discretize the target hidden representation and convert it into a specific speech token. The quantization process includes the following steps: calculating similarity, the vector quantizer selects a code vector from the codebook that is closest to the target hidden representation. Specifically, it calculates the distance (e.g., Euclidean distance or cosine similarity) between each code vector in the codebook and the target hidden representation; it selects the code vector with the smallest distance, considering it as the vector most similar to the target hidden representation. This closest code vector best represents the features of the current semantic information.

[0062] For example, after finding the code vector with the minimum distance, the index value of that code vector in the codebook can be obtained. This index value serves as the discrete speech token, which is a quantization result of the current semantic information, facilitating further processing during speech generation.

[0063] It should be noted that discrete speech tokens represent unique speech features of a specific semantic component within the target text. A sequence of multiple discrete speech tokens can be used for subsequent speech generation, ensuring that the generated speech accurately conveys the meaning of the original text. These discrete speech tokens can be input into a vocoder to predict specific speech signals, thus progressively achieving a complete text-to-speech conversion process.

[0064] The above implementation method maps semantic information into discrete speech tokens through the cooperation of convolutional encoders and vector quantizers. The discrete speech tokens retain the features of semantic information and realize discretized representation through codebook, providing a unified discrete coding foundation for subsequent speech generation.

[0065] Optionally, the target large language model is obtained by: acquiring a training set, which includes several input texts; labeling the training set to obtain corresponding labeling results, wherein the labeling results of the training set include the speech stream corresponding to each input text; and training the initial large language model using the training set and the labeling results to obtain the target large language model.

[0066] Specifically, the labeled results corresponding to the training set can be used as the labels for the input data of that group. Then, each group of training sets carrying labels is input into the initial diffusion model for supervised learning. When the training termination condition is met, such as when the number of training times reaches the number threshold or the output accuracy of the model reaches the accuracy threshold, the training ends and the target large language model is obtained after training is completed.

[0067] In this embodiment, the training set and annotation results can be input into an initial diffusion model for supervised learning, thereby training a target large language model. This allows for the output of the speech stream corresponding to the text based on the target large language model.

[0068] Based on the above embodiments, the initial large language model is trained using the training set and annotation results to obtain the target large language model, including: concatenating the input text and speech stream on the time axis to obtain concatenated training data; and training the initial large language model based on the concatenated training data to obtain the target large language model.

[0069] It should be understood that by concatenating the input text and speech stream along a timeline, the changes and characteristics of the target object at different times can be reflected more comprehensively, increasing data diversity and helping to expand the model's generalization ability. Furthermore, the concatenated annotation results combine information from multiple time points, reducing the error of data annotation at a single moment, improving annotation accuracy, and thus enhancing the model's training performance. It can also, to some extent, mitigate noise interference that may exist in a single input text or speech stream, improving the model's adaptability to complex scenarios.

[0070] The above embodiments enhance data quality and diversity during the training of the target large language model, thereby improving the model's generalization ability and practical application effectiveness.

[0071] Optionally, after obtaining the target large language model, the process further includes: iteratively training the target large language model to extract data features and calculating the loss function; iteratively training the loss function using a preset method with the aim of reducing the value of the loss function until the expected threshold is met; and obtaining the iterative target large language model based on the loss function after iterative training.

[0072] Understandably, in order to train a target large language model with higher accuracy, the target large language model can be trained iteratively to continuously reduce the loss function until the loss function meets the expected threshold. Then, a more accurate speech stream corresponding to the text input can be obtained based on the iterated target large language model.

[0073] It should be noted that this application does not limit the above-mentioned preset method and expected threshold. For example, the preset method can be gradient descent algorithm, batch gradient descent algorithm, stochastic gradient descent algorithm, etc. This application uses gradient descent algorithm as an example for explanation.

[0074] The purpose of the gradient descent algorithm is to find the minimum value of the loss function, or to converge to the minimum value, through iteration. Geometrically speaking, gradient descent occurs where the gradient decreases most rapidly along the opposite direction of the vector where the function's change increases most rapidly, thus making it easier to find the function's minimum. Based on this, in this embodiment, the gradient descent algorithm can be used to iteratively train the target large language model, continuously reducing the loss function and thereby reducing the error in the calculation results.

[0075] In this embodiment, the loss function is continuously reduced by iteratively training the denoising model using the gradient descent algorithm to obtain the iterative target large language model. Then, a more accurate speech stream corresponding to the text input can be obtained based on the iterative target large language model.

[0076] Please see Figure 3 , Figure 3This is a schematic block diagram of a text-to-speech apparatus according to an embodiment of this application. The text-to-speech apparatus can be configured in a server to perform the aforementioned text-to-speech method.

[0077] like Figure 4 As shown, the text-to-speech device 200 includes: an acquisition module 201, a processing module 202, a determination module 203, and a conversion module 204.

[0078] The acquisition module 201 is used to acquire the target text and determine the semantic information corresponding to the target text;

[0079] Processing module 202 is used to process the semantic information through the speech segmenter in the target large language model to obtain the discrete speech token corresponding to the target text, wherein the discrete speech token includes the encoded information corresponding to the semantic information;

[0080] The determining module 203 is used to determine the location information corresponding to the discrete speech token based on the encoded information; and to predict the discrete speech token by using the vocoder in the target large language model to obtain the speech signal corresponding to the discrete speech token.

[0081] The conversion module 204 is used to sort the speech signal according to the location information to obtain the target speech stream corresponding to the target text.

[0082] The determining module 203 is further configured to perform preprocessing operations on the target text to obtain preprocessed target text, wherein the preprocessing operations include word segmentation, normalization, and stop word removal; analyze the text structure information of the preprocessed target text, and determine the semantic information corresponding to the target text based on the text structure information.

[0083] Processing module 202 is further configured to acquire a preset codebook and determine the target hidden representation corresponding to the semantic information through the convolutional encoder; determine the code vector with the smallest distance from the code vectors in the preset codebook to the target hidden representation through the vector quantizer, and determine the index of the code vector with the smallest distance as the discrete speech token corresponding to the target text.

[0084] The acquisition module 201 is further configured to acquire a training set, which includes several input texts; to annotate the training set to obtain corresponding annotation results, wherein the annotation results of the training set include the speech stream corresponding to each input text; and to train an initial large language model using the training set and the annotation results to obtain the target large language model.

[0085] The acquisition module 201 is further configured to concatenate the input text and the speech stream on the time axis to obtain concatenated training data; and to train the initial large language model based on the concatenated training data to obtain the target large language model.

[0086] The acquisition module 201 is further configured to iteratively train the target large language model to extract data features and calculate the loss function; iteratively train the loss function using a preset method with the aim of reducing the value of the loss function until the expected threshold is met; and obtain the iterative target large language model based on the loss function after iterative training.

[0087] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the above-described apparatus and its modules and units can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0088] The methods and apparatus of this application can be used in a wide variety of general-purpose or special-purpose computing system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer terminal devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0089] For example, the above-described method and apparatus can be implemented as a computer program, which can be used in, for example... Figure 4 It runs on the computer device shown.

[0090] Please see Figure 4 , Figure 4 This is a schematic diagram of a computer device provided in an embodiment of this application. The computer device may be a server.

[0091] like Figure 4 As shown, the computer device includes a processor, memory, and network interface connected via a system bus, wherein the memory may include volatile storage media, non-volatile storage media, and internal memory.

[0092] Non-volatile storage media can store operating systems and computer programs. These computer programs include program instructions that, when executed, cause the processor to perform any text-to-speech method.

[0093] The processor provides computing and control capabilities, supporting the operation of the entire computer device.

[0094] Internal memory provides an environment for the execution of computer programs stored on non-volatile storage media. When executed by a processor, the computer program enables the processor to perform any method of text-to-speech conversion.

[0095] This network interface is used for network communication, such as sending assigned tasks. Those skilled in the art will understand that the structure of this computer device is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0096] It should be understood that the processor can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among these, a general-purpose processor can be a microprocessor or any conventional processor.

[0097] In some embodiments, the processor is used to acquire target text and determine the semantic information corresponding to the target text; process the semantic information using a speech segmenter in the target large language model to obtain a discrete speech token corresponding to the target text, wherein the discrete speech token includes encoding information corresponding to the semantic information; determine the position information corresponding to the discrete speech token based on the encoding information; predict the discrete speech token using a vocoder in the target large language model to obtain a speech signal corresponding to the discrete speech token; and sort the speech signal according to the position information to obtain the target speech stream corresponding to the target text.

[0098] In some embodiments, the processor is further configured to perform preprocessing operations on the target text to obtain preprocessed target text, wherein the preprocessing operations include word segmentation, normalization, and stop word removal; analyze the text structure information of the preprocessed target text, and determine the semantic information corresponding to the target text based on the text structure information.

[0099] In some embodiments, the processor is further configured to acquire a training set, the training set including several input texts; to annotate the training set to obtain corresponding annotation results, wherein the annotation results of the training set include the speech stream corresponding to each input text; and to train an initial large language model using the training set and the annotation results to obtain the target large language model.

[0100] In some implementations, the processor is further configured to concatenate the input text and the speech stream on a time axis to obtain concatenated training data; and to train the initial large language model based on the concatenated training data to obtain the target large language model.

[0101] In some implementations, the processor is further configured to iteratively train the target large language model to extract data features and calculate a loss function; iteratively train the loss function using a preset method with the aim of reducing the value of the loss function until a predetermined threshold is met; and obtain the iterative target large language model based on the loss function after iterative training.

[0102] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed, implement any of the text-to-speech methods provided in this application.

[0103] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiments, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, SmartMedia Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the computer device.

[0104] Furthermore, the computer-readable storage medium may primarily include a program storage area and a data storage area, wherein the program storage area may store the operating system, at least one application program required for a function, etc.

[0105] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for text-to-speech conversion, characterized in that, The method includes: Obtain the target text and determine the semantic information corresponding to the target text; The semantic information is processed by the speech segmenter in the target large language model to obtain the discrete speech token corresponding to the target text, wherein the discrete speech token includes the encoded information corresponding to the semantic information; Based on the encoded information, the location information corresponding to the discrete speech token is determined; and the speech signal corresponding to the discrete speech token is obtained by predicting the discrete speech token through the vocoder in the target large language model. Used to sort the speech signal according to the location information to obtain the target speech stream corresponding to the target text.

2. The method according to claim 1, characterized in that, Determining the semantic information corresponding to the target text includes: The target text is preprocessed to obtain the preprocessed target text, wherein the preprocessing operations include word segmentation, normalization and stop word removal. The text structure information of the preprocessed target text is analyzed, and the semantic information corresponding to the target text is determined based on the text structure information.

3. The method according to claim 1, characterized in that, The discrete voice token is obtained using the following formula: Where, μ l For the discrete voice token; ||·| |2 The L2 norm is used; VQ is the vector quantizer; h l is for hidden representation; C is for default codebook.

4. The method according to claim 1, characterized in that, The speech segmenter includes a vector quantizer and a convolutional encoder. The process of processing the semantic information using the speech segmenter in the target large language model to obtain discrete speech tokens corresponding to the target text includes: A preset codebook is obtained, and the target hidden representation corresponding to the semantic information is determined by the convolutional encoder; The vector quantizer determines the code vector with the smallest distance from the code vectors in the preset codebook to the target hidden representation, and the index of the code vector with the smallest distance is determined as the discrete speech token corresponding to the target text.

5. The method according to claim 1, characterized in that, The target large language model is obtained through the following method: Obtain a training set, which includes several input texts; The training set is labeled to obtain the corresponding labeling results, wherein the training set labeling results include the speech stream corresponding to each input text; The initial large language model is trained using the training set and the annotation results to obtain the target large language model.

6. The method according to claim 5, characterized in that, The step of training the initial large language model using the training set and the annotation results to obtain the target large language model includes: The input text and the speech stream are concatenated on the time axis to obtain the concatenated training data. The initial large language model is trained based on the spliced ​​training data to obtain the target large language model.

7. The method according to claim 5, characterized in that, After obtaining the target large language model, the process further includes: The target large language model is iteratively trained to extract data features, and the loss function is calculated. The loss function is iteratively trained using a preset method with the aim of reducing the value of the loss function until the expected threshold is met. Based on the loss function after iterative training, the iterative target large language model is obtained.

8. A text-to-speech device, characterized in that, The device includes: The acquisition module is used to acquire the target text and determine the semantic information corresponding to the target text; The determination module is used to process the semantic information through the speech segmenter in the target large language model to obtain the discrete speech token corresponding to the target text, wherein the discrete speech token includes the encoded information corresponding to the semantic information; The determining module is used to determine the location information corresponding to the discrete speech token based on the encoded information; and to predict the discrete speech token by using the vocoder in the target large language model to obtain the speech signal corresponding to the discrete speech token. The conversion module is used to sort the speech signal according to the location information to obtain the target speech stream corresponding to the target text.

9. A computer device, characterized in that, include: A memory and a processor; wherein the memory is connected to the processor for storing a program for the processor to implement the steps of the text-to-speech method as described in any one of claims 1-7 by running the program stored in the memory.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the steps of the text-to-speech method as described in any one of claims 1-7.

Images