Speech generation method and device based on large language model, and storage medium

Abstract

The application discloses a speech generation method and device based on a large language model and a storage medium. The method comprises the following steps: obtaining historical call text and corresponding historical audio data; inputting the historical call text into a pre-trained text model to obtain semantic text features, and inputting the historical audio data into a pre-trained speech model to obtain speech content features; inputting the semantic text features and the speech content features into a to-be-trained speech synthesis model for fusion, and generating predicted audio data according to the fused features, so that speech content enhancement during training input is realized; training the to-be-trained speech synthesis model according to the predicted audio data and the historical audio data to obtain a trained speech synthesis model; when receiving to-be-converted text, inputting the to-be-converted text into the trained speech synthesis model to generate target audio data corresponding to the to-be-converted text, which is beneficial to improving the authenticity of generated speech.

Description

Technical Field

[0001] This application relates to the field of speech generation technology, and in particular to a speech generation method, device and storage medium based on a large language model. Background Technology

[0002] With the rapid development of digital and artificial intelligence technologies, speech synthesis technology, as an advanced means of converting text information into audible speech, has been widely used in many fields. Early speech synthesis relied on phoneme combinations and statistical models, but the generated speech was often not natural enough. However, with the rise of deep learning, the quality and naturalness of speech synthesis have been greatly improved, enabling it to simulate the voices of different people, and thus it has been widely used in medical, financial and other scenarios.

[0003] However, despite the high degree of naturalness achieved in speech synthesis, significant limitations remain in simulating the complexity and emotional expression of human speech. Existing speech synthesis systems struggle to understand and generate dialogues with depth and nuance, especially those involving emotional changes and contextual understanding. This difficulty in personalizing and mimicking subtle differences in human speech, such as pitch, speech rate, and rhythm, limits the realism and adaptability of the generated speech in certain situations. Consequently, this affects the effectiveness of speech synthesis technology in scenarios that demand high levels of emotional expression and contextual understanding. Summary of the Invention

[0004] The main objective of this application is to provide a speech generation method, device, and storage medium based on a large language model, aiming to solve the technical problem that the speech generated by existing speech synthesis models has limited realism and adaptability.

[0005] To achieve the above objectives, this application proposes a speech generation method based on a large language model, the method comprising:

[0006] Obtain a historical call dataset, which includes: historical call text and corresponding historical audio data;

[0007] The historical call text is input into a pre-trained text model to obtain semantic text features, and the historical audio data is input into a pre-trained speech model to obtain speech content features;

[0008] The semantic text features and the speech content features are input into the speech synthesis model to be trained;

[0009] The semantic text features and the speech content features are fused using the speech synthesis model to be trained, and predicted audio data is generated based on the fused features.

[0010] The speech synthesis model to be trained is trained based on the predicted audio data and the historical audio data to obtain the trained speech synthesis model.

[0011] When receiving text to be converted, the text to be converted is input into the trained speech synthesis model to generate target audio data corresponding to the text to be converted.

[0012] In one embodiment, the speech synthesis model to be trained includes: a feature fusion module and a speech synthesis module;

[0013] The step of fusing the semantic text features and the speech content features through the speech synthesis model to be trained, and generating predicted audio data based on the fused features, includes:

[0014] The feature fusion module obtains the feature weight parameters for the current round, and performs weighted fusion of the speech content features and the semantic text features based on the feature weight parameters for the current round to obtain fused features;

[0015] The speech synthesis module generates predicted audio data for the current round based on the fusion features.

[0016] In one embodiment, before the step of weighted fusion of the speech content features and the semantic text features based on the feature weight parameters of the current round, the following steps are included:

[0017] The semantic text features are subjected to positional encoding enhancement processing to obtain encoded enhanced semantic text features;

[0018] Accordingly, the step of weighted fusion of the speech content features and the semantic text features based on the feature weight parameters of the current round includes:

[0019] The speech content features and the encoded enhanced semantic text features are weighted and fused based on the weight parameters of the current round to obtain fused features.

[0020] In one embodiment, the speech synthesis module includes a variational autoencoder and a decoder, and the step of generating the predicted audio data for the current round by the speech synthesis module based on the fusion features includes:

[0021] The variational autoencoder obtains the corresponding feature distribution parameters based on the fused features;

[0022] The decoder generates a corresponding speech waveform based on the feature distribution parameters, and uses the speech waveform as the predicted audio data for the current round.

[0023] In one embodiment, the step of training the speech synthesis model to be trained based on the predicted audio data and the historical audio data to obtain the trained speech synthesis model includes:

[0024] Based on the predicted audio data of the current round and the historical audio data, the loss value of the current round is calculated according to a preset loss function;

[0025] When the loss value does not meet the preset loss threshold condition, the model parameters of the speech synthesis model to be trained are adjusted according to the loss value to obtain the speech synthesis model to be trained in the next round, and the feature weight parameters of the current round are updated to obtain the feature weight parameters of the next round.

[0026] Returning to the step of inputting the semantic text features and the speech content features into the speech synthesis model to be trained, until the loss value meets the preset loss threshold condition, the trained speech synthesis model is obtained.

[0027] In one embodiment, the step of inputting the text to be converted into the trained speech synthesis model to generate target audio data corresponding to the text to be converted when receiving the text to be converted includes:

[0028] Upon receiving the text to be converted, the text features to be converted are extracted using the pre-trained text model to obtain the text features to be converted.

[0029] The features of the text to be converted are input into the trained speech synthesis model to obtain the target audio data corresponding to the text to be converted.

[0030] In one embodiment, the step of inputting the historical call text into a pre-trained text model to obtain semantic text features includes:

[0031] The historical call text is input into a pre-trained text model, and the semantic information of the historical call text is captured through the self-attention mechanism in the pre-trained text model to obtain text features containing deep semantics as semantic text features.

[0032] The pre-trained text model is a text feature extraction model built on the Transformer architecture and pre-trained on a large-scale text dataset.

[0033] In one embodiment, the step of inputting the historical audio data into a pre-trained speech model to obtain speech content features includes:

[0034] The historical audio data is input into a pre-trained speech model, and the speech content features are extracted from the historical audio data through the pre-trained speech model to obtain the corresponding speech content features.

[0035] The speech content features include: speech speed, speech rhythm, and speech prosody;

[0036] The pre-trained speech model is a speech feature extraction model trained using a self-supervised learning method on an unlabeled audio dataset.

[0037] Furthermore, to achieve the above objectives, this application also proposes a speech generation device based on a large language model, the device comprising: a memory, a processor, and a speech generation program based on a large language model stored in the memory and executable on the processor, the speech generation program based on a large language model being configured to implement the steps of the speech generation method based on a large language model as described above.

[0038] Furthermore, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a speech generation program based on a large language model is stored. When the speech generation program based on the large language model is executed by a processor, it implements the steps of the speech generation method based on the large language model as described above.

[0039] This application discloses a speech generation method based on a large language model. The method involves acquiring a historical call dataset, which includes historical call text and corresponding historical audio data; inputting the historical call text into a pre-trained text model to obtain semantic text features, and inputting the historical audio data into a pre-trained speech model to obtain speech content features; inputting the semantic text features and speech content features into a speech synthesis model to be trained; fusing the semantic text features and speech content features through the speech synthesis model to be trained, and generating predicted audio data based on the fused features; training the speech synthesis model to be trained based on the predicted audio data and historical audio data to obtain a trained speech synthesis model; and when receiving text to be converted, inputting the text to be converted into the trained speech synthesis model to generate the target audio data corresponding to the text to be converted.

[0040] This application fully utilizes text and audio information from historical call datasets, combining pre-trained text and speech models to extract semantic text features and speech content features respectively, thereby enhancing the content of the training input features for the speech synthesis model. This enables the speech synthesis model to more comprehensively learn the language understanding and speech expression capabilities required for speech generation, improving the realism of the generated speech. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart illustrating the first embodiment of the speech generation method based on a large language model according to this application;

[0044] Figure 2 This is a flowchart illustrating the second embodiment of the speech generation method based on a large language model in this application;

[0045] Figure 3 This is a flowchart illustrating the third embodiment of the speech generation method based on a large language model in this application;

[0046] Figure 4 This is a schematic diagram of the training process of the speech synthesis model in this application;

[0047] Figure 5 This is a schematic diagram of the speech generation device based on a large language model according to this application.

[0048] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0050] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0051] This application provides a speech generation method based on a large language model, referencing... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the speech generation method based on a large language model according to this application. In this embodiment, the method includes steps S10 to S60:

[0052] Step S10: Obtain the historical call dataset, which includes historical call text and corresponding historical audio data.

[0053] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or speech processing server. It can also be other electronic devices capable of accessing the speech generation model, and this embodiment does not limit it. The following uses a speech generation device based on a large language model (hereinafter referred to as "device") as an example to describe the various embodiments of this application.

[0054] Understandably, the historical call dataset can include call data collected in real-world scenarios. For example, in a financial debt collection scenario, the historical call dataset can include the actual audio of customer service personnel making debt collection calls to different customers, as well as the converted text of the calls.

[0055] It should be understood that the historical call texts in this historical call dataset can record the call content in written form, preserving the utterance sequence, key information, and semantic details completely; while the corresponding historical audio data can contain rich speech features, such as timbre, intonation, and volume. Using this historical call dataset as a training dataset for subsequent speech synthesis models can lay a solid foundation for subsequent model training, ensuring that the model can learn on real and reliable corpora.

[0056] Step S20: Input the historical call text into a pre-trained text model to obtain semantic text features, and input the historical audio data into a pre-trained speech model to obtain speech content features.

[0057] It should be understood that this pre-trained text model can employ a large-scale Chinese text model designed for Chinese use. This model is built upon the Transformer architecture and pre-trained on a large-scale text training dataset, enabling it to fully understand and process the complex semantics of Chinese, transforming the input text into rich semantic representations. During the pre-training process of this text model, model tuning can be achieved by optimizing the cross-entropy loss function. The tuning formula is as follows:

[0058]

[0059] in, y represents the model parameters of the text model. i For the real label, p(y) i | ) represents the model's predicted probability.

[0060] By inputting historical call texts into this pre-trained text model, the model can capture semantic information of the text through a self-attention mechanism, such as implicit information, contextual dependence, and emotional coloring, thereby obtaining a text representation with rich deep semantics, i.e., semantic text features, which provides an accurate text base for subsequent speech generation.

[0061] It should also be noted that this pre-trained speech model can be a self-supervised speech pre-trained model. During the pre-training process of this speech model, it can learn from unlabeled audio datasets using a self-supervised learning approach, specifically based on a contrastive loss function. The formula for this contrastive loss function is as follows:

[0062]

[0063] Among them, z i and z j It is the feature representation of positive sample pairs, T is the temperature parameter, and (sim(·)) is the similarity parameter.

[0064] Pre-trained audio models can extract key content features from audio, such as phonemes, rhythm, and prosody, thereby constructing speech feature vectors to reflect the textual information of the audio, providing strong support for establishing the correspondence between audio and text.

[0065] In a specific implementation, the device can input the historical call text into a pre-trained text model, and use the self-attention mechanism in the pre-trained text model to capture semantic information of the historical call text, thereby obtaining text features containing deep semantics as semantic text features; and input the historical audio data into a pre-trained speech model, and use the pre-trained speech model to extract speech content features from the historical audio data, thereby obtaining the corresponding speech content features.

[0066] Step S30: Input the semantic text features and the speech content features into the speech synthesis model to be trained.

[0067] It should be understood that the speech synthesis model to be trained can be a text-based conditional variational coding speech codec model, which combines a variational autoencoder (VAE) and a conditional generation model. It can take text as conditional input during the speech encoding process, thereby generating corresponding speech based on the text content.

[0068] It should be noted that this speech codec model can learn the distribution features of both audio and text simultaneously, ensuring not only the correspondence between the generated speech and the input text, but also the ability to handle the learning of various accents and timbres.

[0069] Step S40: The semantic text features and the speech content features are fused using the speech synthesis model to be trained, and predicted audio data is generated based on the fused features.

[0070] It should be understood that after acquiring the aforementioned semantic text features and speech content features, the speech synthesis model to be trained can first perform feature fusion. By means of, for example, weighted averaging, the semantic text features and speech content features can be integrated to generate a comprehensive input representation containing rich and balanced information, namely, the fused features.

[0071] Next, the text-based conditional variational coding speech codec model can convert the fused features of the above input into audio signals layer by layer according to the fused feature mapping rules, and output the predicted audio data.

[0072] Step S50: Train the speech synthesis model to be trained based on the predicted audio data and the historical audio data to obtain the trained speech synthesis model.

[0073] It should be understood that this historical audio data can be used as a training target for the speech synthesis model to determine whether the speech synthesis model obtained after each iteration has achieved the training objective.

[0074] Understandably, in each iteration, the predicted audio data output by the speech synthesis model can be compared with the real historical audio data, and the error signal can be backpropagated based on the loss function and with the help of optimization algorithms (such as gradient descent), so that the parameters of each layer in the speech synthesis model can be dynamically adjusted to achieve parameter tuning of the speech synthesis model.

[0075] This process is repeated iteratively until the loss function converges. The speech synthesis model completes its training by continuously learning the distribution features of audio and text and constantly approximating the rules of real speech generation. Ultimately, it can generate a personalized, high-quality speech synthesis model that retains semantic content.

[0076] Step S60: When receiving the text to be converted, input the text to be converted into the trained speech synthesis model to generate the target audio data corresponding to the text to be converted.

[0077] It should be understood that the text to be converted can be the original text that the user needs to generate speech in a real-world application scenario, which can be input by the user on the device's interactive interface.

[0078] It should be noted that the above-trained speech synthesis model uses pre-trained parameters and architecture. It can take the text features of the text to be converted as input, convert the semantic information contained in the text into the corresponding speech expression, and output target audio data that is semantically consistent with the text to be converted and has natural and fluent speech characteristics, thereby completing the speech generation task and realizing the conversion from text to sound.

[0079] This embodiment utilizes a powerful pre-trained text model to extract textual information with deep semantic meaning, and combines it with a self-supervised pre-trained speech model to extract important content features of speech, thereby strengthening the connection between text and speech. This further enhances the speech generation model's ability to learn personalized expressive information of the text within the speaker's context. The pre-trained text model can directly use the input text as input to generate corresponding semantic content, leveraging the latest advantages of large-scale language models to fully understand the semantic depth information in different scenarios. The self-supervised large-scale speech model, serving as the pre-trained speech model, can fully extract the textual content information of the audio as a supplement to the audio-to-text relationship. This allows the subsequent speech synthesis model, after obtaining fused features based on semantic text features and speech content features, to more comprehensively learn the language understanding and speech expression capabilities required for speech generation through a training process based on these fused features, resulting in a higher-quality speech synthesis model.

[0080] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the speech generation method based on a large language model in this application.

[0081] In this embodiment, to specifically illustrate how the speech synthesis model generates audio data, the speech synthesis model can be divided into a feature fusion module and a semantic synthesis model. Therefore, step S40 further includes: steps S401~S402:

[0082] Step S401: Obtain the feature weight parameters of the current round through the feature fusion module, and perform weighted fusion of the speech content features and the semantic text features based on the feature weight parameters of the current round to obtain fused features.

[0083] It should be noted that the feature weight parameter can be a weight parameter that is dynamically adjusted as the model is trained during the generation of fused features, including the concatenation weight corresponding to the speech content features and the concatenation weight corresponding to the semantic text features.

[0084] Understandably, during the training of a speech synthesis model, this feature weight parameter can also be regarded as a dynamic parameter, thus allowing it to be trained synchronously with the model parameters.

[0085] In each round, the feature fusion module obtains the feature weight parameters for the current round, and then fuses the speech content features and semantic text based on a weighted average formula to obtain fused features. The weighted average formula is as follows:

[0086]

[0087] In the formula, It is the fused feature vector. , These are dynamically adjusted weight parameters. It is a semantic text feature. It is a feature of speech content.

[0088] In addition, to further improve the richness of features input to the speech codec model, relative positional encoding with a self-attention mechanism can be introduced to enhance text features. Therefore, before weighted fusion of the speech content features and the semantic text features based on the feature weight parameters of the current round, positional encoding enhancement processing can be performed on the semantic text features to obtain encoded enhanced semantic text features.

[0089] It should be noted that this positional encoding enhancement process can be a relative positional encoding method that introduces a self-attention mechanism to enhance text encoding. This method takes into account the relative positional information in the text sequence, making the text encoding richer. This helps to learn and utilize the semantic space more effectively in subsequent speech encoding and decoding models, that is, to better understand the order relationship and relative position of words in the text, thereby improving the generalization ability and generation effect of the model.

[0090] It should also be noted that positional encoding enhancement can be achieved by optimizing the learnable positional weights in the attention mechanism, and the corresponding formula is:

[0091]

[0092] Where Q, K, and V are the query, key, and value vectors, respectively, and d k is the dimension of the key vector.

[0093] In the specific implementation, the feature fusion module can first perform positional encoding enhancement on the semantic text features to obtain encoded enhanced semantic text features; then, based on the weight parameters of the current round, it can perform weighted fusion of the speech content features and the encoded enhanced semantic text features to obtain fused features.

[0094] Step S402: The speech synthesis module generates the predicted audio data for the current round based on the fusion features.

[0095] It should be understood that the speech synthesis module can be constructed based on a text-conditional variational coding speech codec model, which can be divided into a variational autoencoder and a decoder.

[0096] Therefore, step S402 specifically includes: steps S4021~S4022:

[0097] Step S4021: Obtain the corresponding feature distribution parameters based on the fused features using the variational autoencoder.

[0098] It should be understood that a variational autoencoder can process fused features through an encoder network to learn the corresponding feature distribution parameters, namely the mean and variance of the latent variables. The encoder network structure can be a multilayer perceptron, etc., and the learning process can be implemented through forward propagation computation. This embodiment does not limit the specific structure of the variational autoencoder.

[0099] It should be noted that the encoder of a variational autoencoder can map the input features to a probability distribution in the latent space, rather than a fixed point, thereby introducing uncertainty and increasing the diversity of the generated data. Furthermore, since the input features in this embodiment are fused features, this variational autoencoder can simultaneously learn the distribution features of both audio and text.

[0100] Next, the variational autoencoder can sample in the latent space based on the mean and variance of the latent variables to obtain the specific values ​​of the latent variables, thus obtaining the feature distribution parameters.

[0101] One commonly used sampling method is reparameterization technique sampling, which allows for efficient gradient calculation during backpropagation and solves the problem of non-differentiability during the sampling process.

[0102] Step S4022: The decoder generates the corresponding speech waveform according to the feature distribution parameters, and uses the speech waveform as the predicted audio data for the current round.

[0103] It should be understood that the aforementioned feature distribution parameters, i.e. the sampled latent variables, can be input into the decoder, enabling the decoder to generate the corresponding speech waveform based on the latent variables.

[0104] The decoder can be structured as an autoregressive model, a generative adversarial network, etc., to reconstruct speech data that matches the input fusion features based on latent variables.

[0105] In its implementation, the decoder can generate high-quality speech waveforms by learning the mapping relationship between latent variables and speech waveforms. These generated speech waveforms can be used as the predicted audio data for the current round, and can be used to calculate loss values ​​by comparing them with historical audio data corresponding to the fused features, thus enabling iterative training of the model.

[0106] In this embodiment, a weighted averaging technique is used to fuse speech content features and semantic text features during feature fusion. The weight parameters can be obtained through synchronous training with the speech codec model's parameters, thereby generating fused features containing rich and balanced feature information. This allows the subsequent speech codec model to receive more comprehensive features, resulting in higher naturalness and human-like qualities in the generated speech, achieving seamless text-to-speech conversion. Furthermore, a self-attention mechanism for relative positional encoding is introduced during feature fusion. By considering the relative positional information in the text sequence, enhanced semantic text features are obtained, making the semantic text features richer. This, in turn, helps the subsequent speech codec model learn and utilize the semantic space more effectively, improving the model's generalization ability and generation performance.

[0107] Based on the first and second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and second embodiments described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the speech generation method based on a large language model in this application.

[0108] In this embodiment, to further explain the training process of the speech synthesis model, step S50 includes: steps S501~S503:

[0109] Step S501: Based on the predicted audio data of the current round and the historical audio data, calculate the loss value of the current round according to the preset loss function.

[0110] It should be understood that training the speech synthesis model to be trained is an iterative optimization process. By continuously comparing the predicted audio data with historical audio data, the model parameters can be adjusted in each round to improve the quality of speech generation.

[0111] It should be noted that a preset loss function can be used to calculate the difference between the predicted audio data of the current round and the historical audio data, thus obtaining the loss value for the current round. This preset loss function can combine multiple loss terms, such as mean squared error (MSE) and KL divergence loss, to comprehensively measure the gap between the generated speech and the real speech.

[0112] For example, the preset loss function can be the loss function of a variational autoencoder, including a reconstruction loss function and a KL divergence loss. The formula for the preset loss function can be expressed as:

[0113]

[0114] in, It is the reconstruction loss value. This is the KL divergence loss value. The balancing parameter can be used to balance the weights of the reconstruction loss and the KL divergence loss, controlling the trade-off between reconstruction accuracy and generation diversity in the model.

[0115] It should be understood that the reconstruction loss value measures the difference between the generated speech waveform and the real speech waveform, and is usually calculated using mean squared error (MSE) or mean absolute error (MAE). It reflects the model's performance in reconstructing the details of the speech waveform.

[0116] KL divergence loss is used to ensure that the distribution of latent variables approximates a standard normal distribution, thereby improving the diversity and quality of the generated data. The formula for calculating the KL divergence loss value is as follows:

[0117]

[0118] In the formula, The mean of the latent variables. is the standard deviation of the latent variable.

[0119] Step S502: When the loss value does not meet the preset loss threshold condition, adjust the model parameters of the speech synthesis model to be trained according to the loss value to obtain the speech synthesis model to be trained in the next round, and update the feature weight parameters of the current round to obtain the feature weight parameters of the next round.

[0120] It should be understood that the preset loss threshold can be an indicator set based on experience or experimentation to determine whether the training of the speech codec model has achieved the expected results.

[0121] After each iteration of training, the predicted audio data output by the semantic codec model and the historical audio data (original speech waveform or original speech spectrum) corresponding to the fusion features of that round can be input into the aforementioned preset loss function to calculate the loss value after the current iteration round.

[0122] If the loss value is greater than the preset loss threshold, it can be determined that the speech codec model has not reached the training objective. Based on the loss value of the current round, the model parameters can be adjusted using optimization algorithms (such as gradient descent) to minimize the loss value. At the same time, the feature weight parameters used for feature fusion can be adjusted to reflect the relative importance of semantic text features and speech content features in the new training round.

[0123] After adjusting the parameters and updating the feature weights as described above, we can obtain the speech synthesis model to be trained in the next round and the feature weight parameters for the next round.

[0124] Step S503: Return to the step of inputting the semantic text features and the speech content features into the speech synthesis model to be trained, until the loss value meets the preset loss threshold condition, and obtain the trained speech synthesis model.

[0125] It should be understood that the updated speech synthesis model to be trained and the feature weight parameters are re-input into the training process, starting from the next round of inputting semantic text features and speech content features into the speech synthesis model to be trained, and repeating the entire training iteration process.

[0126] The training process continues until the loss value of the current round meets the preset loss threshold. At this point, the speech synthesis model is considered to have achieved the expected training effect, and the obtained speech synthesis model is the trained speech synthesis model.

[0127] Here you can also refer to Figure 4 The training process of the speech synthesis model in the speech generation method based on a large language model in this application is described. Figure 4 This is a schematic diagram of the training process for the speech synthesis model in this application.

[0128] Depend on Figure 4 As can be seen, firstly, call data collected in real-world scenarios can be used as the training dataset, which may include: real historical call text and real audio data.

[0129] Input the historical call text into the pre-trained text model. This pre-trained text model is a powerful Chinese pre-trained model that can extract deep semantic features from the input text, capture complex semantic information such as emotional color and context dependence contained in the text, and thus obtain semantic information features and text features (which can be integrated into semantic text features).

[0130] Historical audio data is input into a pre-trained speech model, which is a self-supervised learning model based on a large-scale unlabeled audio dataset. It can extract key content features of speech, such as phonemes, rhythm, and prosody, and obtain speech content features. These speech content features can reflect the text information in the audio, providing strong support for establishing the correspondence between audio and text.

[0131] Next, a feature fusion module can be set before the original speech codec model. This feature fusion module can perform weighted feature fusion on the extracted semantic text features and speech content features to obtain fused features.

[0132] The fused features are then used as input to a speech encoding and decoding model (autoencoder speech generation model), which is based on a text conditional variational coding architecture and can encode and decode the fused features to generate predicted speech waveforms.

[0133] Finally, the predicted speech waveform can be compared with the actual spectral feature results (historical audio data): the actual spectral feature results can be input into the speech codec model and converted into the actual speech waveform; then, the loss value is calculated by inputting the predicted speech waveform and the actual speech waveform into the preset loss function to obtain the loss value of the current round.

[0134] If the loss value does not meet the preset loss threshold, the model parameters are adjusted based on the loss value. Simultaneously, the feature weight parameters are updated to reflect the relative importance of semantic text features and speech content features in the new training rounds.

[0135] The updated model and feature weight parameters are re-invested in training. The steps of feature extraction, fusion, model training, and loss calculation are repeated until the loss value meets the preset loss threshold condition. Then the training is complete, and the trained speech synthesis model can be obtained.

[0136] Furthermore, to illustrate in detail how to apply the trained speech synthesis model, step S60 specifically includes: steps S601~S602:

[0137] Step S601: Upon receiving the text to be converted, extract text features from the text to be converted using the pre-trained text model to obtain the text features to be converted.

[0138] It should be understood that when receiving text to be converted, the text can first be processed by a pre-trained text model to convert it into a high-dimensional text feature vector, which encodes the core semantics of the text.

[0139] Step S602: Input the features of the text to be converted into the trained speech synthesis model to obtain the target audio data corresponding to the text to be converted.

[0140] Understandably, since the trained speech synthesis model has already learned how to convert text features into speech features during training, and has already integrated the knowledge of the pre-trained speech model into its own model parameters, it can utilize its internal feature fusion module and speech synthesis module, combined with the feature weight parameters learned during training, to convert the input text features to be converted into speech features. Then, the decoder generates the corresponding speech waveform, ultimately outputting high-quality target audio data.

[0141] This embodiment employs an iterative training method, enabling the speech synthesis model (speech codec model) to gradually learn how to better integrate semantic text features and speech content features. This results in a high-quality trained speech synthesis model suitable for various application scenarios, including speech synthesis, speech recognition, and speech-text interaction. Furthermore, since the knowledge provided by the pre-trained speech model is internalized by the trained speech synthesis model, it is no longer involved in the application phase. When the device receives text to be converted, it can input the pre-trained text model to obtain the features of the text to be converted, and then input these features into the trained speech synthesis model to obtain the target audio data corresponding to the text to be converted. This helps improve the efficiency and convenience of speech generation.

[0142] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the speech generation method based on the large language model in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0143] This application also provides a speech generation device based on a large language model. The speech generation device based on a large language model includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the speech generation method based on the large language model in the first embodiment described above.

[0144] The following is for reference. Figure 5 , Figure 5 This is a schematic diagram of the speech generation device based on a large language model according to this application. The speech generation device based on a large language model in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The speech generation device based on a large language model shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0145] like Figure 5As shown, the speech generation device based on a large language model may include a processor 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the speech generation device based on the large language model. The processor 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the large language model-based speech generation device to exchange data wirelessly or via wired communication with other devices. Although a large language model-based speech generation device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.

[0146] The speech generation device based on a large language model provided in this application, employing the speech generation method based on a large language model in the above embodiments, can solve the technical problems of speech generation based on a large language model. Compared with the prior art, the beneficial effects of the speech generation device based on a large language model provided in this application are the same as the beneficial effects of the speech generation method based on a large language model provided in the above embodiments, and other technical features in this speech generation device based on a large language model are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0147] This application also provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the speech generation method based on a large language model in the above embodiments.

[0148] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0149] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described speech generation method based on a large language model, thereby solving the technical problems of the speech generation method based on a large language model. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the speech generation method based on a large language model provided in the above embodiments, and will not be repeated here.

[0150] The above-mentioned embodiment numbers are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. They are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application's specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of this application.

Claims

1. A speech generation method based on a large language model, characterized in that, The method includes: Obtain a historical call dataset, which includes: historical call text and corresponding historical audio data; The historical call text is input into a pre-trained text model to obtain semantic text features, and the historical audio data is input into a pre-trained speech model to obtain speech content features. The speech content features include phonemes, rhythm, and prosody. The semantic text features include implicit information, context dependence, and emotional coloring. The semantic text features and the speech content features are input into the speech synthesis model to be trained. The speech synthesis model to be trained includes a feature fusion module and a speech synthesis module. The semantic text features and the speech content features are fused using the speech synthesis model to be trained, and predicted audio data is generated based on the fused features, including: The feature fusion module obtains the feature weight parameters for the current round, performs positional encoding enhancement on the semantic text features to obtain encoded enhanced semantic text features, and performs weighted fusion on the speech content features and the encoded enhanced semantic text features based on the weight parameters for the current round to obtain fused features. The feature weight parameters include the concatenation weights corresponding to the speech content features and the semantic text features. The speech synthesis module generates predicted audio data for the current round based on the fusion features, including: the speech synthesis module includes a variational autoencoder and a decoder; the feature weight parameters and the model parameters of the speech synthesis model are trained synchronously; the variational autoencoder obtains corresponding feature distribution parameters and a probability distribution that maps the input features to the latent space based on the fusion features; the feature distribution parameters include the mean and variance of the latent variables; the decoder generates a corresponding speech waveform based on the feature distribution parameters, and uses the speech waveform as the predicted audio data for the current round. The speech synthesis model to be trained is trained based on the predicted audio data and the historical audio data to obtain the trained speech synthesis model. When receiving text to be converted, the text to be converted is input into the trained speech synthesis model to generate target audio data corresponding to the text to be converted.

2. The method as described in claim 1, characterized in that, The step of training the speech synthesis model to be trained based on the predicted audio data and the historical audio data to obtain the trained speech synthesis model includes: Based on the predicted audio data of the current round and the historical audio data, the loss value of the current round is calculated according to a preset loss function; When the loss value does not meet the preset loss threshold condition, the model parameters of the speech synthesis model to be trained are adjusted according to the loss value to obtain the speech synthesis model to be trained in the next round, and the feature weight parameters of the current round are updated to obtain the feature weight parameters of the next round. Returning to the step of inputting the semantic text features and the speech content features into the speech synthesis model to be trained, until the loss value meets the preset loss threshold condition, the trained speech synthesis model is obtained.

3. The method as described in claim 1, characterized in that, The step of receiving text to be converted, inputting the text to be converted into the trained speech synthesis model, and generating target audio data corresponding to the text to be converted includes: Upon receiving the text to be converted, the text features to be converted are extracted using the pre-trained text model to obtain the text features to be converted. The features of the text to be converted are input into the trained speech synthesis model to obtain the target audio data corresponding to the text to be converted.

4. The method as described in claim 1, characterized in that, The step of inputting the historical call text into a pre-trained text model to obtain semantic text features includes: The historical call text is input into a pre-trained text model, and the semantic information of the historical call text is captured through the self-attention mechanism in the pre-trained text model to obtain text features containing deep semantics as semantic text features. The pre-trained text model is a text feature extraction model built on the Transformer architecture and pre-trained on a large-scale text dataset.

5. The method as described in claim 1, characterized in that, The pre-trained speech model is a speech feature extraction model trained using a self-supervised learning method on an unlabeled audio dataset.

6. A speech generation device based on a large language model, characterized in that, The speech generation device based on a large language model includes: a memory, a processor, and a speech generation program based on a large language model stored in the memory and executable on the processor. When the speech generation program based on a large language model is executed by the processor, it implements the steps of the speech generation method based on a large language model as described in any one of claims 1 to 5.

7. A storage medium, characterized in that, The storage medium stores a speech generation program based on a large language model, which, when executed by a processor, implements the steps of the speech generation method based on a large language model as described in any one of claims 1 to 5.

Images