A speech synthesis method, apparatus, electronic device, and storage medium

Abstract

The application provides a speech synthesis method and device, electronic equipment and storage medium, the method comprises the following steps: obtaining target text corresponding to the synthesized speech audio and input speech; based on the speech pause style of the input speech, pause prediction is performed on the target text to obtain a pause prediction result corresponding to the target text; based on the pause prediction result, the target text is segmented to obtain a plurality of segmented subtexts; based on a preset conversion method, the plurality of subtexts are converted to obtain a plurality of text units; the plurality of text units are combined according to a preset length to obtain a plurality of text sequences; the plurality of text sequences are input into a speech cloning model in a preset order, and the speech audio corresponding to the target text is output; wherein the speech cloning model is used to output the corresponding audio according to the input text. The method can effectively improve the naturalness of speech synthesis and meet the individual needs of users.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a speech synthesis method, apparatus, electronic device, and storage medium in the field of computers. Background Technology

[0002] With the advancement of technology, speech synthesis technology has been widely used in more and more electronic devices. Speech synthesis is a technology that converts text into human speech. Its main function is to imitate human speech patterns, generating corresponding speech output from input text, thereby enabling machines to "speak".

[0003] However, current speech synthesis technology produces speech that is not natural enough and cannot adequately meet users' personalized needs. Summary of the Invention

[0004] This application provides a speech synthesis method, apparatus, electronic device, and storage medium. The method can effectively improve the naturalness of speech synthesis and meet the personalized needs of users.

[0005] Firstly, a speech synthesis method is provided, comprising: acquiring target text and input speech corresponding to the audio to be synthesized; predicting pauses in the target text based on the pause style of the input speech to obtain a pause prediction result corresponding to the target text; segmenting the target text based on the pause prediction result to obtain multiple sub-texts; performing conversion processing on the multiple sub-texts based on a preset conversion method to obtain multiple text units; combining the multiple text units according to a preset length to obtain multiple text sequences; and inputting the multiple text sequences into a speech cloning model in a preset order to output the audio corresponding to the target text; wherein, the speech cloning model is used to output the corresponding audio based on the input text.

[0006] The above technical solution predicts the pauses in the target text based on the pause styles of the input speech, obtaining the corresponding pause prediction results. Based on these predictions, the target text is processed, and finally, the corresponding audio is generated. By analyzing the pause styles inherent in the input speech and applying them to the speech synthesis of the target text, the generated speech becomes closer to realistic human communication, reducing the mechanical feel and effectively improving the naturalness of the synthesized speech, while also meeting users' personalized needs. Segmenting the target text according to the pause prediction results allows for the division of longer texts into shorter sub-texts. Speech synthesis based on these sub-texts reduces the computational load of the model, enabling faster synthesis and improving efficiency. Using a preset conversion method, multiple sub-texts are converted into multiple text units, which are then combined into a text sequence of preset lengths. This ensures that the data input to the speech cloning model conforms to its input format requirements, facilitating processing and analysis. Using a text sequence of appropriate length as input to the speech cloning model avoids the burden on the speech cloning model caused by inputting too long text at once, and further improves the efficiency of speech synthesis.

[0007] In conjunction with the first aspect, in certain implementations of the first aspect, multiple text sequences are input into a speech cloning model in a preset order to obtain the speech audio corresponding to the target text. This includes: assigning multiple text sequences to parallel computing units in the speech cloning model in a preset order; wherein a single parallel computing unit is responsible for processing the inference task of a single text sequence; for a single text sequence, performing multiple forward inferences and multiple speech samplings on the assigned text sequence through a single parallel computing unit until the completion status of the inference task of the text sequence meets a first preset condition; when it is determined that the completion status of the inference tasks of multiple text sequences all meets the first preset condition, merging the multiple inference results output by the parallel computing units to obtain a merged inference result; and generating the speech audio corresponding to the target text based on the merged inference result.

[0008] The above technical solution distributes multiple text sequences to parallel computing units within the speech cloning model, with each unit responsible for processing the inference task of a single text sequence. This parallel processing method significantly improves overall computational efficiency, enabling simultaneous computation on multiple text sequences compared to sequential processing, thus greatly shortening inference time.

[0009] Combining the first aspect and the above implementation methods, in some implementation methods of the first aspect, pause prediction is performed on the target text based on the speech pause style of the input speech to obtain the pause prediction result corresponding to the target text, including: extracting the embedding tensor corresponding to the target text; wherein the embedding tensor contains the pause information corresponding to the target text; extracting the feature tensor corresponding to the speech pause style of the input speech; wherein the dimension of the feature tensor is the same as the dimension of the embedding tensor; fusing the embedding tensor and the feature tensor to obtain a text fusion feature containing speech pause style and pause information; and obtaining the pause prediction result corresponding to the target text based on the text fusion feature.

[0010] The above technical solution extracts the embedding tensor containing pause information corresponding to the target text and the feature tensor corresponding to the pause style of the input speech, and then fuses the two to obtain text fusion features. The pause prediction result obtained based on the text fusion features takes into account both the potential pause positions of the target text itself (such as pause positions determined by grammatical structure) and the actual speech pause styles exhibited in the input speech, making the obtained pause prediction results more comprehensive and accurate, and closer to the pause patterns of real speech.

[0011] Combining the first aspect and the above implementation methods, in some implementation methods of the first aspect, pause prediction is performed on the target text based on the pause style of the input speech to obtain the pause prediction result corresponding to the target text, including: inputting the input speech and the target text into the trained pause prediction model and outputting the pause prediction result corresponding to the target text.

[0012] Combining the first aspect and the above implementation methods, in some implementation methods of the first aspect, the pause prediction model is trained in the following way: obtaining sample speech, the text corresponding to the sample speech, and the pause position label corresponding to the sample speech; constructing an initial pause prediction model; iteratively training the initial pause prediction model based on the sample speech, the text corresponding to the sample speech, and the pause position label corresponding to the sample speech to obtain the trained pause prediction model.

[0013] Combining the first aspect and the above implementation methods, in some implementations of the first aspect, the initial pause prediction model includes a trained text embedding extraction module, an initial speech style extraction module, an initial feature fusion module, and an initial classification network. Based on sample speech, the corresponding text, and pause position labels, the initial pause prediction model is iteratively trained to obtain a trained pause prediction model. This includes: inputting the text corresponding to the sample speech into the trained text embedding extraction module to obtain a sample embedding tensor; wherein the sample embedding tensor includes pause information of the text corresponding to the sample speech; inputting the sample speech into the initial speech style extraction module to obtain the pause prediction model of the sample speech... The sample feature tensor corresponding to the speech pause style is generated; the sample embedding tensor and the sample feature tensor are input into the initial feature fusion module to obtain the sample fusion features; the sample fusion features are input into the initial classification network to obtain the pause prediction result of the text corresponding to the sample speech; based on the pause prediction result and the pause position label corresponding to the sample speech, the parameters corresponding to the initial speech style extraction module, the initial feature fusion module, and the initial classification network are adjusted respectively until the pause prediction result output by the initial classification network meets the second preset condition; if the pause prediction result output by the initial classification network meets the second preset condition, the pause prediction model training is considered complete.

[0014] The above technical solution extracts the sample embedding tensor corresponding to the text of the sample speech through the trained text embedding extraction module, which helps the pause prediction model better understand the text. The initial speech style extraction module extracts the sample feature tensor corresponding to the sample speech, allowing the pause prediction model to better consider the pause styles of real speech, thus making the predicted pause results closer to the actual pause patterns. The initial feature fusion module fuses the sample embedding tensor and the sample feature tensor to obtain sample fusion features containing both text pause information and speech pause style, enabling the pause prediction model to comprehensively consider both text pause information and speech pause style, avoiding the limitations of considering only a single factor in prediction. Through continuous iterative training, the pause prediction model can gradually correct its parameters, making the prediction results increasingly closer to the actual pause positions, thereby continuously optimizing the performance of the pause prediction model. A second preset condition is set, requiring the pause prediction results output by the initial classification network to meet this condition before the model training is considered complete, ensuring that the trained pause prediction model has high quality and stability.

[0015] Combining the first aspect and the above implementation methods, in some implementation methods of the first aspect, the text embedding extraction module is trained in the following way: obtaining a text dataset labeled with pause position labels; and adjusting the model parameters of the pre-trained language model based on the text dataset to obtain the trained text embedding extraction module.

[0016] The above technical solution trains the text embedding extraction module and other modules in the pause prediction model separately. This allows the text embedding extraction module to focus on learning the semantics, grammatical structure, and pause-related features of the text itself. It avoids interference from the training process of other modules during the training of the text embedding extraction module, and can more flexibly optimize the training process of the text embedding extraction module, thereby improving the training efficiency and performance of the pause prediction model.

[0017] Secondly, a speech synthesis device is provided, comprising: an acquisition module for acquiring target text and input speech corresponding to the speech audio to be synthesized; a prediction module for predicting pauses in the target text based on the pause style of the input speech, thereby obtaining the corresponding pause prediction result in the target text; a segmentation module for segmenting the target text based on the pause prediction result, thereby obtaining multiple sub-texts; an output module for converting the multiple sub-texts based on a preset conversion method, thereby obtaining multiple text units; combining the multiple text units according to a preset length, thereby obtaining multiple text sequences; and inputting the multiple text sequences into a speech cloning model according to a preset order, thereby outputting the speech audio corresponding to the target text; wherein the speech cloning model is used to output the corresponding audio based on the input text.

[0018] In conjunction with the second aspect, in some implementations of the second aspect, the generation module includes a generation unit, which is specifically used for: allocating multiple text sequences to parallel computing units in the speech cloning model according to a preset order; wherein, a single parallel computing unit is responsible for processing the inference task of a single text sequence; for a single text sequence, performing multiple forward inferences and multiple speech samplings on the allocated text sequence through a single parallel computing unit until the completion status of the inference task of the text sequence meets a first preset condition; when it is determined that the completion status of the inference tasks of multiple text sequences all meets the first preset condition, merging the multiple inference results output by the parallel computing units to obtain a merged inference result; and generating the speech audio corresponding to the target text based on the merged inference result.

[0019] Combining the second aspect and the above implementation methods, in some implementation methods of the second aspect, the prediction module is specifically used for: extracting the embedding tensor corresponding to the target text; wherein the embedding tensor contains the pause information corresponding to the target text; extracting the feature tensor corresponding to the speech pause style of the input speech; wherein the dimension of the feature tensor is the same as the dimension of the embedding tensor; fusing the embedding tensor and the feature tensor to obtain a text fusion feature containing speech pause style and pause information; and obtaining the pause prediction result corresponding to the target text based on the text fusion feature.

[0020] Combining the second aspect and the above implementation methods, in some implementation methods of the second aspect, the prediction module is also specifically used to: input the input speech and target text into the trained pause prediction model, and output the pause prediction result corresponding to the target text.

[0021] In conjunction with the second aspect and the above implementation methods, in some implementation methods of the second aspect, the device further includes a pause prediction model generation module, which is used to: obtain sample speech, text corresponding to the sample speech, and pause position labels corresponding to the sample speech; construct an initial pause prediction model; and iteratively train the initial pause prediction model based on the sample speech, text corresponding to the sample speech, and pause position labels corresponding to the sample speech to obtain a trained pause prediction model.

[0022] Combining the second aspect and the above implementation methods, in some implementation methods of the second aspect, the initial pause prediction model includes a trained text embedding extraction module, an initial speech style extraction module, an initial feature fusion module, and an initial classification network. The pause prediction model generation module includes a generation unit, which is specifically used for: inputting the text corresponding to the sample speech into the trained text embedding extraction module to obtain a sample embedding tensor; wherein, the sample embedding tensor includes pause information of the text corresponding to the sample speech; inputting the sample speech into the initial speech style extraction module to obtain a sample feature tensor corresponding to the speech pause style of the sample speech; inputting the sample embedding tensor and the sample feature tensor into the initial feature fusion module to obtain sample fusion features; inputting the sample fusion features into the initial classification network to obtain the pause prediction result of the text corresponding to the sample speech; based on the pause prediction result and the pause position label corresponding to the sample speech, adjusting the parameters corresponding to the initial speech style extraction module, the initial feature fusion module, and the initial classification network respectively, until the pause prediction result output by the initial classification network meets the second preset condition; when the pause prediction result output by the initial classification network meets the second preset condition, it is determined that the pause prediction model training is complete.

[0023] In conjunction with the second aspect and the above implementation methods, in some implementation methods of the second aspect, the device further includes a text embedding extraction module generation unit, which is specifically used to: obtain a text dataset labeled with pause position tags; and adjust the model parameters of the pre-trained language model based on the text dataset to obtain a trained text embedding extraction module.

[0024] Thirdly, an electronic device is provided, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, causing the electronic device to perform the speech synthesis method in the first aspect and any possible implementation thereof.

[0025] Fourthly, a computer program product is provided, comprising: computer program code, which, when run on a computer, causes the computer to execute the speech synthesis method in the first aspect and any possible implementation thereof.

[0026] Fifthly, a computer-readable storage medium is provided that stores computer program code, which, when executed on a computer, causes the computer to perform the speech synthesis method described in the first aspect and any possible implementation thereof. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of a speech synthesis process in the prior art;

[0028] Figure 2 This is a schematic flowchart of a speech synthesis method provided in an embodiment of this application;

[0029] Figure 3 This is a schematic diagram of the structure of a text embedding and extraction module provided in an embodiment of this application;

[0030] Figure 4 This is a schematic diagram of the structure of a speech style extraction module provided in an embodiment of this application;

[0031] Figure 5 This is a schematic diagram of the structure of a pause prediction model provided in an embodiment of this application;

[0032] Figure 6 This is a schematic diagram of a speech synthesis process provided in an embodiment of this application;

[0033] Figure 7 This is a schematic diagram of the structure of a speech synthesis device provided in an embodiment of this application;

[0034] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0036] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0037] With the advancement of technology, speech synthesis technology has been widely used in more and more electronic devices. Speech synthesis is a technology that converts text into human speech. Its main function is to imitate human speech patterns, generating corresponding speech output from input text, thereby enabling machines to "speak".

[0038] For example, the speech synthesis process in the prior art can be as follows: Figure 1 As shown, the long text to be synthesized is first obtained, then the long text is divided into several short texts according to the punctuation marks, and then the several short texts are combined according to the set batch_size to obtain multiple batches.

[0039] Here, `batch_size` refers to the number of tasks input into the speech cloning model for parallel inference processing. For example, suppose a long text is split into n short texts, if... Figure 1 If the batch_size is set to N, then n short texts can be divided into n / N batches.

[0040] Furthermore, if n / N is an integer, then the text can be divided into n / N batches; if n / N is not an integer, then n / N can be rounded up to obtain an integer number of batches. For example, if batch_size is set to 2 and the number of short texts is 9, then n / N is calculated to be 4.5. Rounding 4.5 up, then the text can be divided into 5 batches. After dividing the short texts into n / N batches, the speech cloning model can perform inference on the n / N batches simultaneously, i.e., parallel inference, to obtain the speech segment corresponding to each short text in the n / N batches.

[0041] After obtaining the audio segments corresponding to each short text in n / N batches, the audio segments corresponding to the short texts in each batch can be concatenated. For example, the audio segments corresponding to short text 1 and short text 2 in batch 1 can be concatenated to obtain the audio of short texts 1 and 2.

[0042] Finally, the audio recordings corresponding to all the short texts can be concatenated to obtain the audio recordings corresponding to the long text.

[0043] However, some long texts may lack punctuation marks. Even after segmenting according to punctuation, there may still be very long sentences, which will affect the quality of speech synthesis and increase inference time. In addition, the speech generated by current speech synthesis technology is not natural enough and cannot well meet the personalized needs of users.

[0044] To address the aforementioned technical problems, embodiments of this application provide a speech synthesis method. The execution subject of this method can be an electronic device, specifically a mobile phone, tablet computer, laptop computer, ultra-mobile personal computer (UMPC), etc. This application does not limit the specific type of electronic device.

[0045] Based on the pause style of the input speech, the embodiments of this application predict the pauses in the target text, which can enable the synthesized speech to simulate the natural language habits of humans and make the generated speech more natural.

[0046] Figure 2 This is a schematic flowchart of a speech synthesis method provided in an embodiment of this application.

[0047] For example, such as Figure 2 As shown, the method 200 includes:

[0048] Step 201: Obtain the target text and input speech corresponding to the audio to be synthesized.

[0049] Step 202: Based on the pause style of the input speech, perform pause prediction on the target text to obtain the pause prediction result corresponding to the target text.

[0050] Step 203: Based on the pause prediction results, the target text is segmented to obtain multiple sub-texts.

[0051] Step 204: Based on the preset conversion method, convert multiple sub-texts to obtain multiple text units.

[0052] Step 205: Combine multiple text units according to a preset length to obtain multiple text sequences.

[0053] Step 206: Input multiple text sequences into the speech cloning model in a preset order and output the speech audio corresponding to the target text.

[0054] The speech cloning model is used to output the corresponding audio based on the input text.

[0055] In this embodiment, pauses in the target text are predicted based on the pause style of the input speech, resulting in a pause prediction result for the target text. Based on this pause prediction result, the target text is processed, and finally, the corresponding audio is generated. By analyzing the pause style contained in the input speech and applying it to the speech synthesis of the target text, the generated speech can be closer to realistic human communication, reducing the mechanical feel and effectively improving the naturalness of the speech synthesis, while also meeting the personalized needs of users. Segmenting the target text according to the pause prediction result can divide a long text into multiple shorter sub-texts. Speech synthesis based on these sub-texts reduces the computational load of the model, allowing for faster speech synthesis and improved efficiency. By converting multiple sub-texts into multiple lexical text units and then combining them into a text sequence of preset length, the data input to the speech cloning model conforms to the input format requirements of the speech cloning model, facilitating its processing and analysis. Using a text sequence of appropriate length as input to the speech cloning model avoids the burden on the speech cloning model caused by inputting too long text at once, and further improves the efficiency of speech synthesis.

[0056] The following is about Figure 2 The implementation methods of each step in the illustrated embodiment are explained in detail below:

[0057] In step 201, the target text refers to the text content that is to be converted into speech using speech synthesis technology. This text can be any form of textual information, such as news reports, novels, instructions, greetings, etc. For example, in voice reading software, the target text could be a picture book story imported by the user. The software can use the parents' pre-recorded voices to synthesize the picture book story into speech, making it convenient for children to hear the picture book story told by their parents through electronic devices.

[0058] The aforementioned input speech refers to the voice information input by the user for specific processing. For example, the input speech could be a sentence spoken by the user through a microphone. Based on this input speech, the user's pause style can be identified. Furthermore, this input speech can also be understood as the speech to be cloned. Based on this input speech, the user's timbre can be determined, thereby synthesizing a synthesized speech with the same timbre as the user.

[0059] It is understandable that different speakers will typically have different pausing styles for the same text. Furthermore, even the same speaker may vary their pausing style depending on the time and context. Based on this, embodiments of this application can predict pauses in the target text based on the pausing style of the input speech, thereby obtaining the corresponding pause prediction result for the target text.

[0060] In step 202, it is understood that everyone has their own unique pausing habits when speaking. For example, some people will pause slightly in the middle of a sentence, while others will pause significantly when listing things. The location, duration, and other characteristics of these pauses constitute the speech pause style.

[0061] For example, the input speech can be analyzed to extract feature information such as the location, duration, and frequency of pauses. For instance, if the input speech is "The weather is nice today, let's go for a walk," and there is a noticeable short pause after "nice," then the location and duration of this pause can be recorded.

[0062] Furthermore, based on the analyzed speech pause styles, pause locations in the target text can be predicted, specifying where pauses should occur and their duration. For example, if the text to be synthesized is "Tomorrow we'll go boating in the park," based on the pause styles of the previously input speech, it can be predicted that there might be pauses after "tomorrow" and "park."

[0063] The above pause prediction result is obtained after predicting pauses in the entire target text. This result clearly indicates the specific location and duration of each pause in the target text. This pause prediction result can be used in subsequent speech synthesis processes to make the synthesized speech more natural.

[0064] For example, the pause prediction result corresponding to the target text can be the pause prediction result corresponding to each character in the target text, which may include whether a pause is needed after the character, the duration of the pause, etc.

[0065] For example, taking the target text "Hello, how's the weather today?" as an example, the pause prediction result for this target text could be: Hello (short pause), (slightly longer pause), (medium pause) how (no pause) is the weather today (no pause), (no pause) how (no pause) is it (no pause) (relatively long pause). The content within each parenthesis above is the pause prediction result for the corresponding text.

[0066] For example, the semantic information of the target text itself can be used in combination with the pause style of the input speech to obtain the pause prediction result corresponding to the target text.

[0067] In one possible implementation, step 202 may include the following steps S11 to S14:

[0068] S11: Extract the embedding tensor corresponding to the target text.

[0069] The embedding tensor contains pause information corresponding to the target text.

[0070] Understandably, in natural language processing, text usually needs to be converted into a numerical representation that a computer can process, and embedding tensors are one such numerical representation that a computer can process.

[0071] For example, each character or word in the target text can be mapped to a fixed-dimensional vector space, so that each character or word has a corresponding vector representation, and the vectors corresponding to these characters or words can form an embedding tensor.

[0072] For example, assuming a fixed dimension of 100, each character or word is represented as a 100-dimensional vector. If the target text contains multiple characters or words, the vectors corresponding to these characters or words are arranged in a certain order to form a higher-dimensional tensor. If the text has N characters or words, and each character or word corresponds to a D-dimensional vector, then the resulting embedding tensor typically has a dimension of N×D.

[0073] Furthermore, the aforementioned embedding tensor can carry pause information corresponding to the target text.

[0074] For example, a text embedding extraction model can be trained to extract the embedding tensor of the target text. This text embedding extraction model can be a model based on a recurrent neural network.

[0075] First, a large amount of text data from different domains, styles, and sentence structures can be collected and used as training data. Special markers are used to indicate pauses in the text, and this labeled text data is divided into training, validation, and test sets. Then, the text embedding extraction model is constructed, consisting of an input layer, embedding layer, RNN layer, pause prediction layer, and output layer.

[0076] The input layer receives the sequence of target text, where each character or word can be encoded into a vector representation. The embedding layer maps the input encoded vectors to a low-dimensional continuous vector space, generating embedding vectors that can be used as input to subsequent layers. The RNN layer uses a recurrent neural network to process the embedding vector sequence and can capture the sequence information and long-term dependencies in the text, which helps in learning pause information and contextual relationships. The pause prediction layer has an output dimension of 2 (which can indicate whether there is a pause or not) and can use an activation function to convert the output into a probability distribution corresponding to text pauses. The output layer outputs the embedding tensor of the entire text, which contains pause information.

[0077] For pause prediction tasks, a loss function (such as the binary cross-entropy loss function) is used to measure the difference between the predicted pause probability and the actual pause label, and an appropriate optimizer is selected to minimize the loss value corresponding to the loss function.

[0078] The text data from the training set is input into the text embedding extraction model, passing through an input layer, embedding layer, RNN layer, pause prediction layer, and output layer to obtain the predicted pause probabilities and embedding tensors. The predicted pause probabilities are compared with the true pause labels, and the loss value is calculated based on the loss function. The gradient of the loss value with respect to the model parameters is calculated using the backpropagation algorithm. The optimizer updates the model parameters based on the calculated gradient, gradually reducing the loss value. During training, the model's performance is periodically evaluated using a validation set. If the loss on the validation set no longer decreases or overfitting occurs, the parameters of the text embedding extraction model can be left unadjusted. The trained model's performance is evaluated using a test set. If the model's performance meets preset standards, the text embedding extraction model can be considered successfully trained.

[0079] Understandably, after training and evaluation, the text embedding extraction model can directly output an embedding tensor containing pause information when there is new target text input. This embedding tensor can be used for subsequent tasks such as pause prediction and speech synthesis.

[0080] For example, after the model used to extract the embedding tensor from the text has been trained, the target text can be input into the trained text embedding extraction model to obtain the embedding tensor corresponding to the target text, and the embedding tensor contains pause information.

[0081] S12: Extract the feature tensor corresponding to the speech pause style of the input speech.

[0082] The dimension of the feature tensor is the same as the dimension of the embedding tensor.

[0083] As we can understand it, a tensor is a data structure that can be viewed as an extension of vectors and matrices in a high-dimensional space. In deep learning, tensors are the basic units used to store and process data. A feature tensor is a tensor that contains specific feature information. For speech pause styles, the feature tensor stores various feature information related to the speech pause style, such as the numerical representation of pause position, duration, and frequency, as mentioned above.

[0084] To ensure better fusion of the embedding tensor corresponding to the target text and the feature tensor corresponding to the input speech in subsequent processing, the dimensions of the embedding tensor and the feature tensor can be unified.

[0085] Specifically, before extracting the feature tensor corresponding to the input speech, the dimension of the embedding tensor corresponding to the target text can be obtained, and the dimension of the feature tensor can be determined based on the dimension of the embedding tensor, so as to ensure that the dimension of the embedding tensor is the same as the dimension of the feature tensor.

[0086] For example, the input speech is first preprocessed, such as segmenting the input speech into sentences to ensure that each segment is relatively independent and of appropriate length, removing background noise from the input speech, and adjusting the volume of the input speech.

[0087] Then, basic acoustic features and pause-related features are extracted from the input speech. The basic acoustic features may include features such as Mel-Frequency Cepstral Coefficients (MFCC) and Linear Prediction Cepstral Coefficients (LPCC). The pause-related features may include: silence duration, silence location, silence frequency, average pause duration, and pause interval distribution.

[0088] The extracted acoustic features and pause-related features are then concatenated to form a unified feature vector.

[0089] To integrate with the aforementioned embedding tensor, a linear mapping function can be used to expand the spatial dimension of the feature vector obtained from the above combination based on the dimension corresponding to the embedding tensor, resulting in a feature tensor with the same dimension as the embedding tensor.

[0090] Finally, the feature tensor can be normalized to make the scales of different features similar.

[0091] S13: Fuse the embedding tensor and the feature tensor to obtain text fusion features that include speech pause style and pause information.

[0092] Since the dimensions of the embedding tensor and the feature tensor are the same, the elements at the same positions in the embedding tensor and the feature tensor can be added together to obtain a new tensor. Alternatively, weights can be assigned to the embedding tensor and the feature tensor respectively, and then the two tensors can be weighted and summed according to the weights to obtain the fused tensor, which is the text fusion feature mentioned above.

[0093] S14: Based on text fusion features, obtain the pause prediction results corresponding to the target text.

[0094] Understandably, obtaining the pause prediction result corresponding to the target text based on text fusion features can usually be achieved with the help of machine learning or deep learning models.

[0095] For example, a large amount of text data with pause annotations can be collected as both a training and testing set. The annotations can be marking the pause locations within the text. Features corresponding to the text data in the training set are extracted to obtain text fusion features. Based on the model structure corresponding to the machine learning model, the input layer, hidden layer, and output layer of the prediction model are constructed. The input layer takes the text fusion features as input, with the input dimension being the same as the dimension of the text fusion features. The hidden layer is used to learn sequence patterns in the text data. The number of neurons in the output layer can be equal to the number of categories to be predicted, and an activation function can be used to convert the output of the output layer into a probability distribution. The output layer can have two neurons. When the output layer has two neurons, the output results in two ways: "pause" and "no pause." The softmax activation function can be used to map the output of the output layer to obtain two probability values: the probability value corresponding to "pause" and the probability value corresponding to "no pause."

[0096] The output layer can also have only one neuron. When the output layer has only one neuron, the output is a "pause" result. A sigmoid activation function can be used for mapping, and a threshold can be set. The prediction result is determined based on the relationship between the result obtained from the activation function mapping and the set threshold.

[0097] The text fusion features corresponding to the text data in the training set are input into the constructed prediction model. The loss between the prediction result and the ground truth label is calculated. Then, the gradient is calculated using the backpropagation algorithm, and the parameters of the prediction model are updated using an optimizer. This process is repeated until the prediction model converges.

[0098] The target text undergoes the same feature extraction and fusion processing as the training data, transforming it into an input format acceptable to the prediction model. The resulting fused text features are then input into the trained prediction model.

[0099] Taking the output of the prediction model as an example where the result is a "pause," the prediction model outputs the probability that each position in the target text is a pause. A threshold (such as 0.5) can be set to predict positions with a probability greater than the threshold as pauses and positions with a probability less than the threshold as non-pauses, thus obtaining the pause prediction result for the target text.

[0100] The method described above extracts the embedding tensor containing pause information from the target text and the feature tensor corresponding to the pause style of the input speech, and then fuses the two to obtain text fusion features. The pause prediction results obtained based on the text fusion features take into account both the potential pause positions in the target text itself (such as pause positions determined by grammatical structure) and the actual pause styles exhibited in the input speech, making the pause prediction results more comprehensive and accurate, and closer to the pause patterns of real speech.

[0101] To obtain the pause prediction results corresponding to the target text more quickly, a pause prediction model can be trained in advance. The input speech and target text can be input into the pause prediction model to obtain the pause prediction results corresponding to the target text.

[0102] In one possible implementation, based on the pause style of the input speech, pause prediction is performed on the target text to obtain the pause prediction result corresponding to the target text. This includes: inputting the input speech and the target text into a trained pause prediction model and outputting the pause prediction result corresponding to the target text.

[0103] Understandably, the above pause prediction model is a trained machine learning model or deep learning model that can predict the pause locations in the target text based on the input speech and the target text.

[0104] For example, as mentioned above, after obtaining the target text and the input speech, feature tensors corresponding to the speech pause styles can be extracted from the input speech, and embedding tensors can be extracted from the target text. Then, the feature tensors and embedding tensors can be fused to obtain text fusion features. The text fusion features can then be input into the trained pause prediction model to predict the pause status at each position in the text based on the text fusion features, and output the pause prediction results.

[0105] In some embodiments, the target text and input speech can be directly input into the pause prediction model, and the pause prediction result corresponding to the target text can be output.

[0106] One possible implementation is that the pause prediction model is trained by: obtaining sample speech, the corresponding text, and the pause position label; constructing an initial pause prediction model; and iteratively training the initial pause prediction model based on the sample speech, the corresponding text, and the corresponding pause position label to obtain the trained pause prediction model.

[0107] It is understandable that the sample speech mentioned above refers to a portion of speech data selected from a large amount of real speech data. The sample speech may come from different speakers, be in different language scenarios, and have different accents, etc.

[0108] Each sample speech has a corresponding text content, which accurately records the content expressed by the speech.

[0109] The pause location labels corresponding to the above sample speech refer to the specific locations of the pauses marked in the corresponding text based on the actual pauses in the sample speech.

[0110] The sample speech, the corresponding text, and the pause position labels of the sample speech can form a training set for training the pause prediction model.

[0111] Furthermore, after obtaining the training set, an initial pause prediction model can be constructed. Specifically, a model architecture can be selected first, such as a recurrent neural network model, a convolutional neural network model, or an attention-based model. After selecting the model architecture, the parameters of the initial pause prediction model can be initialized, for example, by randomly initializing the model parameters, i.e., assigning random values ​​to the model parameters.

[0112] Then, the obtained sample speech, corresponding text, and pause position labels are input into the initial pause prediction model to extract and process features from the input speech and text, and attempt to predict the pause positions in the text.

[0113] After the initial pause prediction model outputs its pause prediction result, the difference between the pause position predicted by the initial pause prediction model and the actual pause position label is determined, and the loss value is calculated based on the defined loss function. Based on the calculated loss value, the backpropagation algorithm is used to calculate the gradient of the loss value with respect to the model parameters of the initial pause prediction model; where the gradient indicates in which direction the model parameters are adjusted to reduce the loss. Then, an optimizer (such as stochastic gradient descent SGD, Adagrad, Adadelta, Adam, etc.) is used to update the parameters of the initial pause prediction model based on the calculated gradient.

[0114] The initial pause prediction model is trained iteratively, repeating the process of inputting data, calculating loss, backpropagation, and updating parameters. In each iteration, the initial pause prediction model updates its parameters based on the new gradients, gradually bringing its predictions closer to the actual pause positions. After multiple iterations, when the loss value of the initial pause prediction model no longer decreases significantly or reaches the preset number of training epochs, the initial pause prediction model is considered to have converged, indicating that training is complete.

[0115] Understandably, a trained pause prediction model can directly predict pauses in target text based on the input speech.

[0116] In some embodiments, the trained pause prediction model may include a text embedding extraction module (i.e., the text embedding extraction model described above), a speech style extraction module, a feature fusion module, and a classification network.

[0117] The text embedding extraction module is used to extract the embedding tensor corresponding to the target text; the speech style extraction module is used to extract the feature vector corresponding to the speech pause style of the input speech; the feature fusion module is used to fuse the embedding tensor and the feature vector to obtain the fused feature; and the classification network is used to classify whether there is a pause at each position in the target text based on the received fused feature, that is, to output the pause prediction result corresponding to the target text.

[0118] It is understandable that, during the training of the above pause prediction model, the individual modules in the model can be trained separately, or the individual modules in the model can be trained together.

[0119] The text embedding extraction module aims to extract rich and meaningful feature representations from text, which need to accurately reflect the semantics, syntax, and other information of the text. To learn general and accurate text features, it needs to be trained on a large amount of diverse text data.

[0120] The speech style extraction module primarily focuses on features such as pause styles in speech. Although speech data exhibits a degree of diversity, its style features are relatively concentrated and patterned. In practical applications, common speech pause style patterns can be learned using a sufficient number of speech samples.

[0121] The feature fusion module integrates text embedding features and speech style features to generate a more comprehensive feature representation. Its training focuses on learning how to effectively fuse two different types of features, without relying on large amounts of data for training.

[0122] The task of the classification network is to predict the classification of pause locations based on the fused features. Since the text embedding and speech style extraction modules already provide relatively effective feature representations, the classification network can learn how to make classification decisions using relatively little data.

[0123] Based on the above, it is clear that the amount of data required to train the text embedding extraction module far exceeds the amount of data required to train the speech style extraction module, feature fusion module, and classification network. Therefore, it is advisable to train the text embedding extraction module first, followed by the speech style extraction module, feature fusion module, and classification network.

[0124] For example, the above-mentioned training pause prediction module can be divided into two stages. The first stage is to train the text embedding extraction module. After the text embedding extraction module is trained, the second stage of training is started, namely, training the speech style extraction module, feature fusion module and classification network.

[0125] In some embodiments, the text embedding extraction module can be trained based on text data with pause labels.

[0126] In one possible implementation, the text embedding extraction module is trained by: obtaining a text dataset labeled with pause position tags; and adjusting the model parameters of the pre-trained language model based on the text dataset to obtain the trained text embedding extraction module.

[0127] Specifically, a large amount of text data is first collected, and professionals annotate the pause positions in the text data. For example, special symbols can be used to mark pause positions in the text, or a label sequence corresponding to the text length can be generated, where each element in the sequence indicates whether a pause exists at the corresponding position. Then, the text data is segmented into words according to the word segmentation method of a pre-trained language model (such as BERT), and the segmented text and corresponding pause labels are organized to construct a format suitable for model input.

[0128] Use tools provided by deep learning frameworks (such as PyTorch and TensorFlow) to load pre-trained language models (such as BERT). Based on this pre-trained language model, make some modifications to enable it to output embedding tensors with pause information. Specifically, you can perform additional processing on the output layer of the pre-trained language model, such as adding a fully connected layer to predict the pause probability at each position in the text.

[0129] Choose a suitable loss function (such as the binary cross-entropy loss function) to measure the difference between the pause positions predicted by the pre-trained language model and the actual pause position labels, and calculate the corresponding loss value. Then select a suitable optimizer, and update the parameters of the pre-trained language model based on the calculated loss value.

[0130] The pre-trained language model is trained iteratively. During each iteration, the pre-trained language model updates its parameters based on the calculated loss value, gradually bringing its output closer to the actual pause location labels. After multiple iterations, when the loss value of the pre-trained language model no longer decreases significantly or reaches the preset number of training epochs, the pre-trained language model is considered to have converged, and training is complete.

[0131] It is understandable that if the pre-trained language model is trained, the trained text embedding extraction module can be obtained.

[0132] For example, the process of outputting the pause prediction results corresponding to the text based on the trained text embedding extraction module can be as follows: Figure 3 As shown, inputting text into the trained text embedding extraction module yields the corresponding embedding tensor. This embedding tensor is then input into the first classification network to obtain the pause prediction result for each character in the text.

[0133] The text embedding extraction module outputs the embedding tensor corresponding to the text, while the first classification network determines the pause prediction result for each character in the text based on the embedding tensor.

[0134] It is understandable that if the first classification network is a binary classification network, that is, the number of neurons in the output layer of the first classification network is 2, then the output of the first classification network is the probability that each character in the text belongs to "pause" and the probability that it does not belong to "non-pause".

[0135] For example, assuming the input sentence length is 100 and the hidden layer feature dimension of the model is 312, the text embedding extraction module can output a 100×312 embedding tensor, and finally the first classification network can obtain a 100×1 tensor.

[0136] It's understandable that the sentence length of 100 in the above input indicates that the text sequence input into the model contains 100 words, or 100 units after word segmentation, tokenization, and other processing. The hidden layer feature dimension of 312 in the above model means that when processing the input text, the hidden layer of the model limits the dimension of the generated feature vector for each input unit or each word to 312. That is, during text processing, the model can extract a 312-dimensional feature representation for each word.

[0137] The text embedding extraction module processes the input sentence of length 100, combining the 312-dimensional feature vectors of each word to form a 100×312 two-dimensional tensor; where the first dimension 100 corresponds to the number of words in the sentence (i.e., the sentence length), and the second dimension 312 corresponds to the feature vector dimension of each word.

[0138] The first classification network is used to classify the obtained embedding tensor. The first classification network takes a 100×312 embedding tensor as input, performs binary classification prediction on each position in the sentence, and finally outputs the "pause" probability value and "non-pause" probability value corresponding to each position in the sentence, indicating the probability that the position belongs to a certain category.

[0139] After processing by the first classification network described above, a 100×1 tensor is obtained. The first dimension 100 of this tensor still corresponds to the number of words in the sentence, and the second dimension 1 represents the probability value of each position belonging to a certain category after being predicted by the classification model. In other words, in this 100×1 tensor, each element is a value between 0 and 1, representing the probability that the corresponding position in the sentence belongs to a certain category (i.e., the probability of belonging to "pause" or "non-pause").

[0140] For example, to convert the output of the first classification network into probability values, the softmax activation function can be used. The softmax activation function maps any real number to the range of 0 to 1, which perfectly matches the range of probability values.

[0141] The above method trains the text embedding extraction module and other modules in the pause prediction model separately. This allows the text embedding extraction module to focus on learning the semantics, grammatical structure, and pause-related features of the text itself. It avoids interference from the training process of other modules during the training of the text embedding extraction module, and can more flexibly optimize the training process of the text embedding extraction module, thereby improving the training efficiency and performance of the pause prediction model.

[0142] Furthermore, after obtaining the trained text embedding extraction module, the second stage of training of the pause prediction model can be carried out, namely, training the speech style extraction module, feature fusion module, and classification network.

[0143] Understandably, the purpose of the second stage of training is to modify the embedding tensor with pause information extracted in the first stage based on the pause style in the input speech, so as to obtain a pause result that is more consistent with the pause style of the input speech.

[0144] In one possible implementation, the initial pause prediction model includes a trained text embedding extraction module, an initial speech style extraction module, an initial feature fusion module, and an initial classification network. The process of training the pause prediction model may include the following steps S21 to S24:

[0145] S21: Input the text corresponding to the sample speech into the trained text embedding extraction module to obtain the sample embedding tensor.

[0146] The sample embedding tensor includes pause information of the text corresponding to the sample speech.

[0147] It is understandable that the text embedding extraction module trained above is the text embedding extraction model described earlier. Specifically, the pause information of the text corresponding to the sample speech can be the pause prediction result for each character in the text corresponding to the sample speech.

[0148] S22: Input the sample speech into the initial speech style extraction module to obtain the sample feature tensor corresponding to the speech pause style of the sample speech.

[0149] For example, such as Figure 4 As shown, the initial speech style extraction module may include an fbank extractor 401, a convolutional layer 402, a GRU layer (gated recurrent unit) 403, and an attention module 404.

[0150] The fbank extractor 401 is used to convert the preprocessed speech frame into fbank features; fbank features are an important feature representation in the field of speech signal processing. They are a set of feature vectors obtained by filtering the spectrum of the speech signal, which can effectively characterize the energy distribution of the speech signal in different frequency bands.

[0151] Convolutional layer 402 is used to extract local patterns and features from the fbank features.

[0152] GRU layer 403 is a variant of a recurrent neural network (RNN) that is capable of processing sequential data and capturing the temporal dependencies of speech signals.

[0153] Attention module 404 is used to output the style feature vector corresponding to the speech, that is, the speech style feature vector mentioned above.

[0154] After determining the structure of the initial speech style extraction module, the initial parameters of the initial speech style extraction module can be set, and then the sample speech can be input into the initial speech style extraction module to obtain the sample feature tensor corresponding to the speech pause style of the sample speech.

[0155] It is understandable that the dimensions of the sample feature tensor and the sample embedding tensor are the same.

[0156] S23: Input the sample embedding tensor and sample feature tensor into the initial feature fusion module to obtain the sample fusion features.

[0157] For example, design a module that can fuse sample feature tensors and sample embedding tensors. For instance, you can add the elements of the sample embedding tensor and the sample feature tensor at the same position to obtain a new tensor; or you can use a more complex fusion method, which can assign weights to different parts based on the correlation between speech and text, and then perform a weighted sum to obtain the fused sample features.

[0158] S24: Input the sample fusion features into the initial classification network to obtain the pause prediction results of the text corresponding to the sample speech.

[0159] For example, a classification network can be constructed to predict pause locations. This classification network can be a multilayer perceptron, a convolutional neural network, or other network architecture suitable for classification tasks. The number of nodes in the output layer of this classification network is determined by the number of pause categories and the type of activation function selected. For example, if it is a binary classification (predicting the probability of "pause" and the probability of "no pause"), and the activation function is a softmax activation function, then the output layer has 2 nodes. The output of this classification network can contain the pause result for each character in the text corresponding to the sample speech.

[0160] S25: Based on the pause prediction results of the text corresponding to the sample speech and the pause position labels corresponding to the sample speech, adjust the parameters corresponding to the initial speech style extraction module, the initial feature fusion module, and the initial classification network respectively until the pause prediction results output by the initial classification network meet the second preset condition.

[0161] Understandably, based on the labeled pause location information and the pause prediction results output by the classification network, an appropriate loss function is used to calculate the loss value. For example, a binary cross-entropy loss function can be used to calculate the loss value.

[0162] Then, the gradients of each parameter in the initial pause prediction model (including the parameters of the initial speech style extraction module, the initial feature fusion module, and the initial classification network) can be calculated using the backpropagation algorithm. Based on the calculated gradients, the parameters of the initial pause prediction model are updated using optimization algorithms (such as stochastic gradient descent, Adam, etc.) to gradually reduce the loss value.

[0163] Repeat the above process of calculating loss, backpropagation, and parameter update, and iterate the training multiple times until the pause prediction result output by the initial classification network meets the second preset condition.

[0164] S26: If the pause prediction results output by the initial classification network meet the second preset condition, the pause prediction model training is considered complete.

[0165] For example, the second preset condition mentioned above can be that the loss value calculated based on the output pause prediction result and the labeled pause position information is less than a preset loss threshold. Alternatively, the second preset condition can be that the similarity between the output pause prediction result and the labeled pause position information is greater than or equal to a preset similarity threshold.

[0166] If the pause prediction results output by the initial classification network meet the second preset condition, it can be determined that the pause prediction model training is complete.

[0167] The text embedding extraction module, after training, extracts the sample embedding tensor corresponding to the text of the sample speech, which helps the pause prediction model better understand the text. The initial speech style extraction module extracts the sample feature tensor corresponding to the sample speech, allowing the pause prediction model to better consider the pause styles of real speech, thus making the predicted pauses closer to the actual pause patterns. The initial feature fusion module fuses the sample embedding tensor and the sample feature tensor to obtain sample fusion features containing both text pause information and speech pause styles. This allows the pause prediction model to comprehensively consider both text pause information and speech pause styles, avoiding the limitations of considering only a single factor in prediction. Through continuous iterative training, the pause prediction model can gradually correct its parameters, making the predicted results increasingly closer to the actual pause positions, thereby continuously optimizing the performance of the pause prediction model. A second preset condition is set, requiring the pause prediction results output by the initial classification network to meet this condition before the model training is considered complete, ensuring that the trained pause prediction model has high quality and stability.

[0168] For example, such as Figure 5 As shown, after the pause prediction model is trained, the target text (i.e., any text in the figure) and the input speech (i.e., the speech in the figure) can be directly input into the pause prediction model.

[0169] The pause prediction model may include a speech style extraction module (i.e., the style encoder in the figure) 501, a text embedding extraction module 502, a feature fusion module 503, and a classification network (i.e., the second classification network in the figure) 504.

[0170] It is understandable that the target text does not need to be identical to the input speech; it can be any piece of text. Figure 5 The asterisk (*) indicates that the parameters of this module are fixed and do not need to be adjusted.

[0171] For example, if the input speech is "I'm going to the park this afternoon, but I don't know if the weather will be good," then it can be determined that there is a clear pause at the position of the " / ". If the target text is "I'm going to the library tomorrow morning, but I'm not sure if it will be open," then the input speech and target text are input into the pause prediction model, and the speech style extraction module of the pause prediction model analyzes the input speech.

[0172] The style encoder 501 can extract acoustic features such as duration, pitch, and volume variations from the input speech, as well as pause style features such as the location, duration, and frequency of pauses. For example, it was found that the average duration of pauses in the input speech was 0.5 seconds, and pauses mainly occurred at the boundaries of phrases and sentences.

[0173] The text embedding extraction module 502 can process the target text, i.e., any text, converting each character in the text into an embedding vector. These vectors contain the semantic and syntactic information of the text. For example, the sentence "I'm going to the library tomorrow morning, but I'm not sure if it's open" is converted into a series of embedding vectors, each vector corresponding to a character.

[0174] The feature fusion module 503 can fuse the speech pause style features obtained by the speech style extraction module and the text features obtained by the text embedding extraction module.

[0175] Classification Network 504 can use fused features to predict pauses. Based on learned patterns and rules, it determines whether a pause should occur at each position in the target text and outputs the pause prediction result for each character in the text. For example, the pause prediction model predicts pause positions such as "I / tomorrow morning / go to the library / but / not sure / if it's open / " based on the pause style of the input speech and the semantic structure of the target text.

[0176] Finally, the pause prediction model outputs the pause prediction results corresponding to the target text, presented in the form of tags. For example, the pause prediction result could be: "I / tomorrow morning / go to the library / but / not sure / if it's open / "; where " / " indicates the predicted pause position.

[0177] In some embodiments, in addition to extracting the speech pause style of the input speech, the timbre features corresponding to the input speech can also be extracted. The extracted timbre features can be used together with the speech pause style features to preprocess the target text, or saved as independent information for input to the subsequent speech cloning model, so as to obtain speech audio with the same timbre as the input speech.

[0178] In step 203, after obtaining the pause prediction result, the target text can be segmented based on the pause prediction result to obtain multiple sub-texts.

[0179] For example, the target text can be initially segmented based on punctuation marks to obtain multiple long sentences. Then, based on the pause prediction results obtained in step 202 above, each long sentence can be segmented to obtain multiple sub-texts.

[0180] In step 204, since electronic devices often cannot directly understand and process natural language text, in order to facilitate processing, the multiple sub-texts obtained from the above segmentation can be converted to obtain text units (i.e., text tokens) that can be understood by electronic devices.

[0181] It is understandable that the aforementioned text units can also be called text tokens. These text units refer to the basic units obtained after processing the original natural language text; these text units can be characters, words, or phrases, etc.

[0182] Based on a preset transformation method, multiple subtexts can be transformed into multiple text units, namely text tokens. Since each token in a text token can be mapped to a vector, electronic devices can perform calculations and processing based on these vectors. By analyzing the relationships and patterns between tokens, they can understand the semantics, syntax, and other information of the text, thereby performing various natural language processing tasks.

[0183] For example, the above-mentioned preset transformation methods can be selected according to actual needs. For instance, the preset transformation method can be the WordPiece segmentation method used by the Bidirectional Encoder Representations from Transformers (BERT) model, or the Byte Pair Encoding Tokenizer (BPE Tokenizer) used by the Generative Pre-trained Transformer (GPT) model.

[0184] In step 205 above, after obtaining multiple text units, the multiple text units (i.e., text tokens) can be combined according to a preset length (i.e., batch_size) to obtain multiple text sequences (batch).

[0185] In some embodiments, to ensure that the lengths of the text token sequences within each batch are consistent, a padding operation can be performed on the text token sequences of each batch. The padding operation involves adding zeros to the ends of shorter sequences to make their lengths match the longest sequence in the batch.

[0186] In step 206 above, it can be understood that the aforementioned speech cloning model is a model capable of outputting corresponding audio based on input text. It has been trained to learn the mapping relationship between text and speech, and can generate corresponding speech audio based on the input text content. When a text sequence is input into the speech cloning model, the model will generate a speech audio segment corresponding to the text sequence based on its internal algorithm and parameters.

[0187] Furthermore, by inputting all the text sequences into the speech cloning model and concatenating the speech audio generated by each text sequence in sequence, the speech audio corresponding to the entire target text can be obtained.

[0188] To achieve more efficient speech synthesis, embodiments of this application may employ a parallel inference method to process multiple text sequences simultaneously.

[0189] In some embodiments, the above-mentioned multiple text units (i.e., text tokens) can be combined according to the number of tasks to be processed in parallel, i.e., batch_size, to obtain multiple text sequences with the same number as batch_size.

[0190] Furthermore, the speech cloning model can include multiple parallel computing units, which can be used to reason about the text sequence and obtain the speech segments corresponding to the text sequence.

[0191] In one possible implementation, the multiple text sequences are input into a speech cloning model in a preset order to obtain the speech audio corresponding to the target text. This includes: assigning the multiple text sequences to parallel computing units in the speech cloning model in a preset order; wherein a single parallel computing unit is responsible for processing the inference task of a single text sequence; for a single text sequence, the assigned text sequence is subjected to multiple forward inferences and multiple speech samplings by a single parallel computing unit until the completion status of the inference task of the text sequence meets a first preset condition; when it is determined that the completion status of the inference tasks of the multiple text sequences all meets the first preset condition, the multiple inference results output by the parallel computing units are merged to obtain a merged inference result; and based on the merged inference result, the speech audio corresponding to the target text is generated.

[0192] Understandably, the speech cloning model contains multiple parallel computing units that can work simultaneously, improving the overall processing speed.

[0193] The above-mentioned preset order can be determined logically based on the text content.

[0194] According to a preset order, multiple prepared text sequences are assigned to different parallel computing units. Each parallel computing unit is only responsible for processing the inference task of one text sequence (i.e., one batch), so that multiple text sequences can be processed by different units at the same time, realizing parallel computing.

[0195] A batch contains multiple text token sequences, and the number of text token sequences is the batch size. Multiple text token sequences in a batch can be processed in parallel.

[0196] Furthermore, the parallel computing unit takes the assigned text sequence as input and feeds it into the speech cloning model for forward propagation computation, and can perform parallel inference on multiple input text token sequences.

[0197] Forward inference, also known as forward propagation, refers to the computational process in which data travels from the input layer of a neural network (i.e., the speech cloning model) through the hidden layer and finally reaches the output layer.

[0198] For example, the speech cloning model uses its internal parameters and structure (such as the calculations of each layer in a neural network) to generate a probability distribution of speech features (such as the pronunciation duration, pitch, timbre, and other related parameters of phonemes) based on the text sequence (batch) composed of the input text token sequence.

[0199] Then, based on the previously generated probability distribution, speech is sampled to obtain the first speech token. Various sampling methods can be used, such as greedy sampling (selecting the value with the highest probability) or random sampling (randomly selecting a value according to the probability distribution). This speech token represents the first step in speech generation.

[0200] The speech tokens obtained from the first sampling are then used as new input or auxiliary information, and forward inference is performed again to generate a new probability distribution. This new probability distribution takes into account the information from the first speech token to generate a more suitable speech token for the next time. Then, sampling is performed again to obtain the next speech token. This process is repeated continuously, with each sampled speech token being related to the previous tokens, gradually building a complete speech sequence.

[0201] By repeating this sampling process multiple times, taking into account the previously sampled speech markers each time, the various parameters of the speech are gradually generated, much like building blocks. After multiple samplings, a series of parameters such as the pronunciation duration, pitch, and timbre of the speech sequence corresponding to the input text sequence can be determined, thereby generating a complete speech sequence.

[0202] In some embodiments, after each loop operation of forward inference and speech sampling is completed in the parallel computing unit, it is determined whether the text token sequence has reached the first preset condition for termination.

[0203] For example, the first preset condition may be that the number of iterations of forward inference and speech sampling is equal to a preset number of iterations, or that a speech unit corresponding to the end symbol is predicted during the inference process. The inference of the text token sequence can be terminated if any one of the first preset conditions is met.

[0204] Understandably, the preset number of loops can be set according to actual needs. Assuming the preset number of loops is set to 100, the parallel computing unit increments the loop count by 1 for each forward inference and speech sampling operation, starting from the beginning of processing the text token sequence. When the count reaches 100, processing of the text token sequence is considered complete. This condition is set to prevent the inference process from looping indefinitely, ensuring the task is completed within a certain range of computing resources and time.

[0205] In text processing, special speech tokens are typically defined to indicate the end of a speech sequence. These terminators are pre-defined and directly accessible to the model, used to identify the end of a text sequence. For example, in speech cloning models based on the Transformer architecture, "[EOS]" (an abbreviation for End Of Sequence) is often used to represent the terminator. When the parallel computing unit predicts the speech tokens corresponding to these terminators during inference and sampling, the processing of the text sequence can be terminated.

[0206] Furthermore, each sequence has a corresponding cache (which can be understood as a temporary storage area) during the generation process to store intermediate results, such as generated speech sampling results and intermediate probability distributions. When a text token sequence reaches one of the above termination conditions, the data in this cache needs to be cleared to avoid interference from the cached data of these terminated sequences, ensuring that the new calculation process is not affected.

[0207] As mentioned earlier, batch size refers to the number of samples processed simultaneously in a single forward computation.

[0208] In some embodiments, if a text sequence satisfies the first preset condition described above, the current batch size is changed during the next forward computation.

[0209] For example, if the current batch size is 5, during the forward inference and speech sampling process, when one of the text sequences meets the first preset condition mentioned above, the current batch size can be changed to 4 for the next forward calculation. This means that in the subsequent calculations, only 4 text sequences are processed at a time.

[0210] Furthermore, the above process is repeated continuously. Whenever a text sequence reaches the termination condition (i.e., the first preset condition mentioned above), its buffer is cleared and the batch size is adjusted, and then the next forward computation is performed. This process continues until all generated sequences reach the termination condition, meaning all sequences have completed their corresponding generation tasks. If the completion status of the inference tasks for all text sequences meets the first preset condition, it indicates that the inference tasks for all text sequences have been completed, and the inference results corresponding to the text sequences can be merged to obtain the merged inference result.

[0211] It is understandable that the above merging inference result could be a complete speech sequence corresponding to the target text.

[0212] Furthermore, the speech cloning model can generate actual speech audio based on the merged inference results using specific algorithms and modules. For example, a diffusion model can be used to convert the inference results into corresponding Mel spectra, and then a vocoder can be used to further convert the Mel spectra into audible speech signals. Ultimately, the speech audio corresponding to the target text is obtained, realizing the conversion from text to speech.

[0213] It is understandable that the audio corresponding to the target text generated above not only has the same timbre as the input speech, but also the same speech pause style as the input speech.

[0214] The method described above assigns multiple text sequences to parallel computing units within the speech cloning model, with each unit responsible for processing the inference task of a single text sequence. This parallel processing approach significantly improves overall computational efficiency. Compared to sequential processing, it can simultaneously perform operations on multiple text sequences, greatly shortening the inference time. The parallel computing units perform repeated forward inference and speech sampling on the assigned text sequences until a first preset condition is met, enabling more efficient inference.

[0215] For example, the speech synthesis method in the above embodiments can be achieved through... Figure 6 The process shown is implemented.

[0216] like Figure 6 As shown, the process includes:

[0217] Step 601: Obtain the long text to be synthesized and the input speech.

[0218] It is understandable that the long text to be synthesized is the aforementioned text. Figure 2 The target text in the above text is the input speech. Figure 2 Input voice in the language.

[0219] Step 602: Input the long text and the input speech into the pause prediction model to obtain the pause prediction result corresponding to the long text.

[0220] It is understandable that the above pause prediction model is the same as the pause prediction model trained earlier, which can output the pause prediction result for each character in a long text.

[0221] Step 603: Based on the pause prediction results, the long text is divided into several short texts.

[0222] Understandably, as mentioned earlier, long texts can be segmented first according to punctuation marks, and then further segmented based on pause prediction results to obtain several short texts.

[0223] Step 604: Combine several short texts into multiple batches according to the set batch_size.

[0224] It's understandable that `batch_size` refers to the number of tasks input into the speech clone model for parallel inference processing. Assuming step 603 above splits the long text into n short texts, if `batch_size` is set to N, then the n short texts can be divided into n / N batches.

[0225] As mentioned earlier, if n / N is not an integer, the number of corresponding batches can be rounded up.

[0226] In some embodiments, the obtained short texts can be first converted into several text tokens that can be recognized by the speech cloning model, and then the several text tokens can be combined according to the batch_size to obtain multiple batches.

[0227] Step 605: Input the obtained multiple batches into the speech cloning model for parallel inference to obtain the speech segment corresponding to each batch.

[0228] For example, multiple batches can be sequentially input into the speech cloning model according to the linguistic logic order to ensure that the final reasoning result conforms to the linguistic logic order.

[0229] Step 606: Concatenate the speech segments corresponding to each batch to obtain the audio corresponding to the long text.

[0230] Figure 7 This is a schematic diagram of the structure of a speech synthesis device provided in an embodiment of this application.

[0231] For example, such as Figure 7 As shown, the device 700 includes:

[0232] The acquisition module 701 is used to acquire the target text and input speech corresponding to the audio to be synthesized.

[0233] The prediction module 702 is used to predict pauses in the target text based on the pause style of the input speech, and obtain the corresponding pause prediction results in the target text.

[0234] The segmentation module 703 is used to segment the target text based on the pause prediction result, and obtain multiple sub-texts after segmentation.

[0235] The generation module 704 is used to transform the multiple sub-texts based on a preset transformation method to obtain multiple text units; combine the multiple text units according to a preset length to obtain multiple text sequences; input the multiple text sequences into a speech cloning model according to a preset order, and output the speech audio corresponding to the target text; wherein, the speech cloning model is used to output the corresponding audio according to the input text.

[0236] In one possible implementation, the generation module includes a generation unit, which is specifically used to: allocate multiple text sequences to parallel computing units in the speech cloning model according to a preset order; wherein, a single parallel computing unit is responsible for processing the inference task of a single text sequence; for the single text sequence, the single parallel computing unit performs multiple forward inferences and multiple speech samplings on the allocated text sequence until the completion status of the inference task of the text sequence meets a first preset condition; when it is determined that the completion status of the inference tasks of multiple text sequences meets the first preset condition, the multiple inference results output by the parallel computing units are merged to obtain a merged inference result; based on the merged inference result, the speech audio corresponding to the target text is generated.

[0237] In one possible implementation, the prediction module is specifically used for: extracting the embedding tensor corresponding to the target text; wherein the embedding tensor contains the pause information corresponding to the target text; extracting the feature tensor corresponding to the speech pause style of the input speech; wherein the dimension of the feature tensor is the same as the dimension of the embedding tensor; fusing the embedding tensor and the feature tensor to obtain a text fusion feature containing speech pause style and pause information; and obtaining the pause prediction result corresponding to the target text based on the text fusion feature.

[0238] In one possible implementation, the prediction module is further specifically used to: input the input speech and target text into the trained pause prediction model, and output the pause prediction result corresponding to the target text.

[0239] Optionally, the device further includes a pause prediction model generation module, which is used to: obtain sample speech, the text corresponding to the sample speech, and the pause position label corresponding to the sample speech; construct an initial pause prediction model; and iteratively train the initial pause prediction model based on the sample speech, the text corresponding to the sample speech, and the pause position label corresponding to the sample speech to obtain a trained pause prediction model.

[0240] In one possible implementation, the initial pause prediction model includes a trained text embedding extraction module, an initial speech style extraction module, an initial feature fusion module, and an initial classification network. The pause prediction model generation module includes a generation unit, which is specifically used for: inputting the text corresponding to the sample speech into the trained text embedding extraction module to obtain a sample embedding tensor; wherein the sample embedding tensor includes pause information of the text corresponding to the sample speech; inputting the sample speech into the initial speech style extraction module to obtain a sample feature tensor corresponding to the speech pause style of the sample speech; inputting the sample embedding tensor and the sample feature tensor into the initial feature fusion module to obtain sample fusion features; inputting the sample fusion features into the initial classification network to obtain the pause prediction result of the text corresponding to the sample speech; adjusting the parameters corresponding to the initial speech style extraction module, the initial feature fusion module, and the initial classification network based on the pause prediction result and the pause position label of the sample speech, until the pause prediction result output by the initial classification network meets a second preset condition; and determining that the pause prediction model training is complete when the pause prediction result output by the initial classification network meets the second preset condition.

[0241] Optionally, the device further includes a text embedding extraction module generation unit, which is specifically used to: obtain a text dataset labeled with pause position tags; and adjust the model parameters of the pre-trained language model based on the text dataset to obtain a trained text embedding extraction module.

[0242] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0243] For example, such as Figure 8 As shown, the electronic device 800 includes a memory 801 and a processor 802. The memory 801 stores executable program code 8011, and the processor 802 is used to call and execute the executable program code 8011 to perform a speech synthesis method.

[0244] Furthermore, embodiments of this application also protect an apparatus that may include a memory and a processor, wherein the memory stores executable program code, and the processor is used to call and execute the executable program code to perform a speech synthesis method provided in embodiments of this application.

[0245] This embodiment can divide the device into functional modules according to the above method example. For example, each module can correspond to a separate function, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0246] When each functional module is divided according to its corresponding function, the device may further include an acquisition module, a prediction module, a segmentation module, and a generation module. It should be noted that all relevant content regarding the steps involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here.

[0247] It should be understood that the apparatus provided in this embodiment is used to execute the above-described speech synthesis method, and therefore can achieve the same effect as the above-described implementation method.

[0248] When using integrated units, the device may include a processing module and a storage module. When applied to an electronic device, the processing module can be used to control and manage the operation of the electronic device. The storage module can be used to support the execution of relevant program code and data by the electronic device.

[0249] The processing module may be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits shown in conjunction with the disclosure of this application. The processor may also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module may be a memory.

[0250] In addition, the apparatus provided in the embodiments of this application may specifically be a chip, component or module. The chip may include a connected processor and a memory. The memory is used to store instructions. When the processor calls and executes the instructions, the chip can execute a speech synthesis method provided in the above embodiments.

[0251] This embodiment also provides a computer-readable storage medium storing computer program code. When the computer program code is run on a computer, the computer executes the aforementioned method steps to implement the speech synthesis method provided in the above embodiment.

[0252] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement a speech synthesis method provided in the above embodiment.

[0253] In this embodiment, the device, computer-readable storage medium, computer program product, or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0254] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0255] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0256] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A speech synthesis method, characterized in that, The method includes: Obtain the target text and input speech corresponding to the audio to be synthesized; Based on the pause style of the input speech, pause prediction is performed on the target text to obtain the pause prediction result corresponding to the target text; Based on the pause prediction results, the target text is segmented to obtain multiple sub-texts; Based on a preset conversion method, the multiple sub-texts are converted to obtain multiple text units; The multiple text units are combined according to a preset length to obtain multiple text sequences; According to a preset order, the multiple text sequences are input into the speech cloning model, and the speech audio corresponding to the target text is output; wherein, the speech cloning model is used to output the corresponding audio according to the input text.

2. The method according to claim 1, characterized in that, The step of inputting the multiple text sequences into the speech cloning model in a preset order to obtain the speech audio corresponding to the target text includes: According to a preset order, the multiple text sequences are assigned to parallel computing units in the speech cloning model; wherein, a single parallel computing unit is responsible for processing the inference task of a single text sequence. For the single text sequence, the single parallel computing unit performs multiple forward inferences and multiple speech samplings on the assigned single text sequence until the completion of the inference task of the text sequence meets the first preset condition. If the completion status of the reasoning tasks of the multiple text sequences meets the first preset condition, the multiple reasoning results output by the parallel computing unit are merged to obtain a merged reasoning result. Based on the merged reasoning results, the corresponding audio of the target text is generated.

3. The method according to claim 1, characterized in that, The step of predicting pauses in the target text based on the pause style of the input speech to obtain the pause prediction result corresponding to the target text includes: Extract the embedding tensor corresponding to the target text; wherein the embedding tensor contains pause information corresponding to the target text; Extract the feature tensor corresponding to the speech pause style of the input speech; wherein, the dimension of the feature tensor is the same as the dimension of the embedding tensor; The embedding tensor and the feature tensor are fused to obtain a text fusion feature that includes the speech pause style and the pause information; Based on the text fusion features, the pause prediction result corresponding to the target text is obtained.

4. The method according to claim 1, characterized in that, The step of predicting pauses in the target text based on the pause style of the input speech to obtain the pause prediction result corresponding to the target text includes: The input speech and the target text are input into the trained pause prediction model, and the pause prediction result corresponding to the target text is output.

5. The method according to claim 4, characterized in that, The pause prediction model was trained in the following way: Obtain sample speech, the text corresponding to the sample speech, and the pause position labels corresponding to the sample speech; Construct an initial pause prediction model; Based on the sample speech, the corresponding text, and the pause position labels, the initial pause prediction model is iteratively trained to obtain the trained pause prediction model.

6. The method according to claim 5, characterized in that, The initial pause prediction model includes a trained text embedding extraction module, an initial speech style extraction module, an initial feature fusion module, and an initial classification network. The initial pause prediction model is iteratively trained based on the sample speech, the corresponding text, and the pause position labels, to obtain the trained pause prediction model, including: The text corresponding to the sample speech is input into the trained text embedding extraction module to obtain a sample embedding tensor; wherein, the sample embedding tensor includes the pause information of the text corresponding to the sample speech; The sample speech is input into the initial speech style extraction module to obtain the sample feature tensor corresponding to the speech pause style of the sample speech; The sample embedding tensor and the sample feature tensor are input into the initial feature fusion module to obtain the sample fusion features; The sample fusion features are input into the initial classification network to obtain the pause prediction results of the text corresponding to the sample speech; Based on the pause prediction results and pause position labels corresponding to the sample speech, the parameters corresponding to the initial speech style extraction module, the initial feature fusion module, and the initial classification network are adjusted respectively until the pause prediction results output by the initial classification network meet the second preset condition. If the pause prediction result output by the initial classification network meets the second preset condition, the pause prediction model is determined to have completed training.

7. The method according to claim 6, characterized in that, The text embedding extraction module is trained in the following way: Obtain a text dataset labeled with pause locations; Based on the text dataset, the model parameters of the pre-trained language model are adjusted to obtain the trained text embedding extraction module.

8. A speech synthesis device, characterized in that, The device includes: The acquisition module is used to acquire the target text and input speech corresponding to the audio to be synthesized. The prediction module is used to predict pauses in the target text based on the pause style of the input speech, and obtain the corresponding pause prediction results in the target text. The segmentation module is used to segment the target text based on the pause prediction result to obtain multiple sub-texts after segmentation; The generation module is used to convert the multiple sub-texts based on a preset conversion method to obtain multiple text units; combine the multiple text units according to a preset length to obtain multiple text sequences; input the multiple text sequences into a speech cloning model according to a preset order, and output the speech audio corresponding to the target text; wherein, the speech cloning model is used to output the corresponding audio according to the input text.

9. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable program code; A processor for calling and running the executable program code from the memory, causing the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1 to 7.

Images