Method, apparatus, device, storage medium and program product for speech generation

Abstract

A method, apparatus, device, storage medium, and program product for speech generation are provided. The method includes processing speech generation information including text and a timbre identifier using a first machine learning model to obtain an intermediate feature representation; determining, based at least on the speech generation information, a target second machine learning model that matches the speech generation information, a model size of the first machine learning model being greater than a model size of the target second machine learning model; processing, using the target second machine learning model, the speech generation information based on the intermediate feature representation to generate a plurality of speech feature representations, the plurality of speech feature representations corresponding to audio for the text, wherein one speech feature representation corresponds to one audio segment in the audio; and generating, based on at least one speech feature representation of the plurality of speech feature representations, at least one audio segment in the audio, the audio having a target timbre indicated by the timbre identifier.

Description

Technical Field

[0001] The examples in this article generally relate to the field of computer science, and in particular to methods, apparatuses, devices, storage media, and program products for speech generation. Background Technology

[0002] Speech generation models can generate audio with a target timbre based on text. Larger-scale generation models usually have strong modeling capabilities, but generating audio segments one by one results in a lot of sequential processing.

[0003] When using a single, small-scale model, it becomes difficult for that model to adapt to different speech generation conditions simultaneously. How to reduce the sequential calls to larger-scale models and ensure that smaller-scale models match the current speech generation conditions is a significant concern within the industry. Summary of the Invention

[0004] In a first aspect of this paper, a speech generation method is provided. The method includes: processing speech generation information using a first machine learning model to obtain intermediate feature representations, wherein the speech generation information includes text and timbre identifiers; determining a target second machine learning model that matches the speech generation information, at least based on the speech generation information, wherein the model size of the first machine learning model is larger than that of the second machine learning model; processing the speech generation information based on the intermediate feature representations using the target second machine learning model to generate multiple speech feature representations, the multiple speech feature representations corresponding to audio for text, one of the speech feature representations corresponding to an audio segment in the audio; and generating at least one audio segment in the audio based on at least one of the multiple speech feature representations, the audio having a target timbre indicated by a timbre identifier.

[0005] In a second aspect of this paper, a speech generation apparatus is provided. This apparatus includes modules for implementing the method of the first aspect.

[0006] In a third aspect of this document, an electronic device is provided. The electronic device includes at least one processor and at least one memory. The at least one memory is coupled to the at least one processor and stores instructions for execution by the at least one processor. When executed by the at least one processor, the instructions cause the electronic device to perform the method of the first aspect.

[0007] In a fourth aspect of this document, a computer-readable storage medium is provided. Computer-executable instructions are stored on the computer-readable storage medium. These computer-executable instructions can be executed by a processor to implement the method of the first aspect.

[0008] In a fifth aspect, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method of the first aspect.

[0009] In this way, a matching small-scale model can be selected based on the current speech generation conditions, and intermediate features provided by the larger-scale model can be used to support the small-scale model in generating multiple speech feature representations. This reduces the serial calls to the larger-scale model and improves speech generation efficiency.

[0010] It should be understood that the content described in this section is not intended to limit the key or important features of the examples in this article, nor is it intended to restrict the scope of the solution. Other features will become readily apparent from the following description. Attached Figure Description

[0011] The above and other features, advantages, and aspects of the various examples herein will become more apparent when taken in conjunction with the accompanying drawings and the following detailed description. In the accompanying drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure can be implemented is shown; Figure 2A An example architecture for speech generation according to some embodiments of this disclosure is shown; Figure 2B An example process for scheduling a second machine learning model by a scheduling module according to some embodiments of the present disclosure is shown; Figure 3 A flowchart illustrating an example process for speech generation according to some embodiments of this disclosure is shown; Figure 4 A schematic structural block diagram of a speech generation apparatus according to some embodiments of the present disclosure is shown; and Figure 5 A block diagram of an electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0012] The examples in the text will now be described in more detail with reference to the accompanying drawings. While some examples are shown in the drawings, it should be understood that solutions can be implemented in various forms and should not be construed as limited to the examples set forth herein. Rather, these examples are provided to provide a more thorough and complete understanding of the solutions. It should be understood that the drawings and examples in this document are for illustrative purposes only and are not intended to limit the scope of protection of the solutions.

[0013] It should be noted that the headings of any section / subsection provided herein are not restrictive. Various examples are described throughout this document, and examples of any type may be included under any section / subsection. Furthermore, examples described in any section / subsection may be combined in any way with any other examples described in the same section / subsection and / or different sections / subsections.

[0014] In the description of the examples in this document, the term "comprising" and similar terms should be understood as open inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "an example" or "the example" should be understood as "at least one example". The term "some examples" should be understood as "at least some examples". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below. It should be noted that, unless explicitly stated otherwise, performing a step in "response to A" does not mean that the step is performed immediately after "A", but may include one or more intermediate steps.

[0015] The examples in this document may involve user data, data acquisition, and / or use. All of these aspects comply with relevant laws, regulations, and rules. In the examples, all data collection, acquisition, processing, manipulation, forwarding, and use are conducted with the user's knowledge and confirmation. Accordingly, when implementing each example, the type, scope of use, and usage scenarios of any data or information that may be involved should be communicated to the user and their authorization obtained through appropriate means, in accordance with relevant laws and regulations. The specific methods of notification and / or authorization can vary depending on the actual situation and application scenario; the scope of the solution is not limited in this regard.

[0016] In this manual and the sample solutions, any processing of personal information will be conducted only under legal grounds (such as obtaining the consent of the data subject or being necessary for the performance of a contract) and will only be carried out within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0017] As used in this paper, the term "model" refers to a model that learns the relationship between inputs and outputs from training data, enabling it to generate corresponding outputs for a given input after training. Model generation can be based on machine learning techniques. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs using multiple layers of processing units. A neural network model is an example of a deep learning-based model. In this paper, "model" may also be referred to as a "machine learning model," "learning model," "machine learning network," or "learning network," and these terms are used interchangeably.

[0018] A neural network is a machine learning network based on deep learning. A neural network can process inputs and provide corresponding outputs. It typically consists of an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications often include many hidden layers, thus increasing the network's depth. The layers of a neural network are connected sequentially, so that the output of the previous layer is provided as the input to the next layer. The input layer receives the inputs to the neural network, while the output layer's output serves as the final output. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the layer above.

[0019] Machine learning typically comprises three phases: training, testing, and application (also known as inference). In the training phase, a given model is trained using a large amount of training data, iteratively updating parameter values ​​until the model can consistently generate inferences that meet the expected goals from the training data. Through training, the model can be considered to have learned the relationship between inputs and outputs (also known as the input-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing phase, test inputs are applied to the trained model to test whether it can provide the correct output, thus determining the model's performance. The testing phase can sometimes be integrated into the training phase. In the application or inference phase, the trained model can be used to process actual model inputs based on the trained parameter values ​​to determine the corresponding model output.

[0020] For ease of understanding, some of the terms used in the embodiments of this disclosure will be explained below.

[0021] The term “speech generation information” as used in this article refers to information used to define the audio to be generated and its generation conditions, which may include text, timbre identifiers, and other conditional information related to the speech generation task.

[0022] The term “text” as used in this article refers to a sequence of characters, words, phonemes, or other data that can characterize language content and is to be converted into audio.

[0023] The term “timbre identifier” as used in this article refers to identifier information used to indicate a target timbre, which may be an index, vector, label, or encoded representation of a reference audio.

[0024] The term "machine learning model" as used in this article refers to a computational model that determines model parameters through training data and performs prediction or generation processes accordingly, and may include neural network models as well as models composed of multiple sub-models.

[0025] The term "first machine learning model" as used in this paper refers to a model that provides intermediate features, performs verification, or generates speech feature representations in the fallback path during speech generation processing, and is larger in scale than the second machine learning model.

[0026] The term “candidate second machine learning model” as used in this paper refers to a candidate model used to generate one or more speech feature representations by combining intermediate feature representations. Multiple candidate second machine learning models may have different applicable conditions, model structures or model states.

[0027] The term “target second machine learning model” as used in this paper refers to a second machine learning model selected from multiple candidate second machine learning models and used for the current speech generation process.

[0028] The term “intermediate feature representation” as used in this paper refers to the internal representation generated by the first machine learning model during the processing of speech generation information, which may include the output of one or more intermediate layers or a fused context vector.

[0029] The term “speech feature representation” as used in this article refers to feature data corresponding to audio or audio segments, which can be continuous vectors, discrete representations, or other data that can be converted into audio by decoding.

[0030] The term "audio segment" as used in this article refers to a portion of audio in the time dimension, which may correspond to a speech feature representation or a set of speech feature representations.

[0031] The term "model size" as used in this article refers to an indicator that characterizes the computational or storage requirements of a model, and can be measured by the number of parameters, the number of layers, the number of hidden dimensions, the amount of computation, or a combination thereof.

[0032] The term “application frequency” as used in this article refers to the number of times the target timbre or corresponding speech generation task is applied within a predetermined time period, the ratio of the number of applications to the duration, or other frequency statistics.

[0033] The term "pre-defined time period" as used in this article refers to a time window used to statistically analyze application frequency, which can be a fixed window or a sliding window.

[0034] The term “attribute information” as used in this article refers to technical attributes that indicate the timbre style associated with a set of speech generation tasks, and may include language category, articulation mode, prosodic category or a combination thereof.

[0035] The term “second machine learning model of the first type” as used in this article refers to a second machine learning model that corresponds to the first timbre set and has a wider range of applicability.

[0036] The term “second type of second machine learning model” as used in this article refers to a second machine learning model corresponding to a second set of timbres, which is a subset of the first set of timbres.

[0037] The term “third type of second machine learning model” as used in this article refers to a candidate model that corresponds to multiple timbres and has a smaller scope of application than the first type of second machine learning model.

[0038] The term “second machine learning model of type four” as used in this article refers to the second machine learning model corresponding to a single timbre.

[0039] The term "available state" as used in this article refers to the state in which the model can be selected and perform the current speech generation process.

[0040] The term "unavailable state" as used in this article refers to a state in which the model has not yet completed training, has not been loaded, is limited by resource conditions, or is not currently involved in the speech generation process.

[0041] The term “threshold” as used in this article refers to a reference value used for comparison or to trigger processing, which can be preset or dynamically adjusted based on validation data, resource status, or runtime statistics.

[0042] The term “fine-tuning” as used in this article refers to the training process of updating at least some parameters using target data based on existing model parameters.

[0043] The term “autoregressive submodel” as used in this article refers to a submodel that generates subsequent representations based on the generated content in sequence.

[0044] The term "diffusion submodel" as used in this paper refers to a submodel that transforms initial variables into a target representation through a multi-step denoising or continuous dynamic process.

[0045] The term “parallel generation” as used in this paper refers to generating two or more speech feature representations in a single model call or the same processing batch.

[0046] The term "floating-point vector" as used in this article refers to a vector consisting of multiple floating-point numbers that can be used to continuously represent the acoustic information of audio.

[0047] The term "verification" as used in this article refers to the process of determining whether the speech feature representations generated by the second machine learning model meet the acceptance criteria.

[0048] The term “denoising trajectory” as used in this paper refers to the state sequence or continuous path that evolves from the initial variable to the target variable in diffusion or flow matching processing.

[0049] The term “log probability density” used in this paper refers to the logarithmic form of the probability density, which characterizes the reliability of candidate speech feature representations under the corresponding model distribution.

[0050] The term "acceptance probability" as used in this paper refers to a value determined based on the probability density relationship between the first machine learning model and the target second machine learning model, used to determine whether a candidate speech feature representation passes verification.

[0051] The term "audio latent vector" used in this paper refers to the feature vector obtained after audio compression and encoding, which can be used as an implementation of speech feature representation.

[0052] The term “audio block” as used in this article refers to a processing unit consisting of the audio latent vectors corresponding to one or more consecutive audio frames.

[0053] The term "Ordinary Differential Equation (ODE)" as used in this paper refers to an equation used to describe the relationship between a continuous state and the independent variable.

[0054] The term “Classifier-Free Guidance (CFG)” used in this paper refers to a guidance method that uses the outputs of conditional and unconditional branches to determine the generated results.

[0055] The term "key-value injection" as used in this article refers to the process of using the intermediate features of the first machine learning model and the internal features of the second machine learning model together for key-value projection.

[0056] The term “Low-Rank Adaptation (LoRA)” used in this paper refers to the process of adapting the model through low-rank parameter updates.

[0057] The term “Real-Time Factor (RTF)” as used in this article refers to the ratio of inference time to the duration of the output audio.

[0058] As mentioned earlier, reducing the serial calls of large-scale models and ensuring that small-scale models match current speech generation conditions is a key concern in the industry. Speech waveforms are typically composed of numerical values ​​arranged in sampling order. Directly predicting each sample point during speech generation would result in a long sequence; therefore, the speech waveform can be compressed and encoded first. The compression result can be a discrete representation or a continuous representation. A continuous representation can consist of multiple floating-point vectors, each corresponding to a certain duration of audio content.

[0059] When generating speech feature representations for larger-scale models one by one, model calculations need to be performed sequentially for each position. If the larger-scale model includes autoregressive and diffusion sub-models, the diffusion sub-model may also perform multiple denoising steps for each position. This process increases the number of serial calls and inference time.

[0060] In view of this, an improved speech generation scheme is proposed. According to this scheme, a first machine learning model processes speech generation information and produces intermediate feature representations. The model application system selects a target second machine learning model from multiple candidate second machine learning models. The target second machine learning model combines the intermediate feature representations to generate multiple speech feature representations. The model application system generates audio with a target timbre based on at least one speech feature representation.

[0061] The above approach allows for different candidate models to be generated under different speech generation conditions. Multiple speech feature representations can be generated in batches by a smaller-scale objective second machine learning model, thereby reducing the number of times the first machine learning model generates speech feature representations one by one. This process shortens the audio generation waiting time and reduces the overhead of sequential inference.

[0062] The following describes various examples of this scheme in further detail with reference to the accompanying drawings.

[0063] Figure 1 A schematic diagram of an example environment 100 for model training and application according to some embodiments of the present disclosure is shown. Figure 1 Example environment 100 illustrates three distinct phases of machine learning model 105, including pre-training phase 102, fine-tuning phase 104, and application phase 106. A testing phase, not shown in the figure, may also occur after pre-training phase 102 or fine-tuning phase 104.

[0064] Although only a single machine learning model 105 is shown in the figure, multiple machine learning models may exist. If machine learning model 105 includes multiple models, these multiple models may include models configured to perform the same or similar tasks, or models configured to perform different tasks.

[0065] Machine learning model 105 can be based on any suitable model architecture, including but not limited to Transformer models, Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Deep Neural Networks (DNNs), and so on. In some examples, machine learning model 130 can be based on a language model (LM), for example. Machine learning model 130 can include language models, Large Language Models (LLMs), Vision-Language Models (VLMs), Multimodal Large Language Models (MLLMs), and so on. Language models, by learning from large corpora, possess question-answering capabilities. Machine learning model 130 can also be based on other suitable models.

[0066] In the pre-training phase 102, the model pre-training system 120 is configured to perform pre-training of the machine learning model 105 using the pre-training dataset 112. At the start of pre-training, the individual components of the machine learning model 105 may have initial parameter values. The pre-training process involves updating the parameter values ​​of the machine learning model 105 to desired values ​​based on the data in the pre-training dataset 112. The pre-training task serves to facilitate the parameter updates of the machine learning model 105.

[0067] In the pre-training phase 102, the machine learning model 105 can learn strong generalization capabilities using a pre-training dataset 112 that includes a large amount of data. After pre-training is complete, the parameter values ​​of the machine learning model 105 have been updated to include the pre-trained parameter values. The pre-trained machine learning model 105 can then be provided to the fine-tuning phase 104, where the model fine-tuning system 130 fine-tunes it for different downstream tasks.

[0068] In the fine-tuning phase 104, the model fine-tuning system 130 further utilizes the training dataset 114 to adjust the parameter values ​​of the machine learning model 105. During fine-tuning, the corresponding training algorithm is also used to update and adjust the parameters of the machine learning model 105. Since the model has already learned a great deal from the training data in the pre-training phase, a satisfactory downstream task model can be obtained using only a small amount of training data in the fine-tuning phase 104.

[0069] The training samples included in training dataset 114 and the pre-training samples included in pre-training dataset 112 can be of the same type. In some cases, pre-training dataset 112 may include multiple pre-training samples for a general scenario, while training dataset 114 may include multiple training samples for a specific scenario or task.

[0070] In some embodiments, a testing phase may be included after the fine-tuning phase 104, where the performance of the machine learning model 105 can be further tested using a test dataset. In some examples, the dataset used in the testing phase is of the same type as that used in the fine-tuning phase.

[0071] In application phase 106, the obtained machine learning model 105, with trained parameter values, can be provided to the model application system 140 for use. In application phase 106, the model application system 140 can utilize the machine learning model 105 to process corresponding model inputs in real-world scenarios and provide corresponding model outputs. In some examples, in a speech generation scenario, the model input may include speech generation information 116, and the model output may include audio 118 generated based on the speech generation information 116. The speech generation information 116 may include text and timbre identifiers, with the timbre identifiers specifying the timbre of the audio 118 to be generated.

[0072] exist Figure 1 In this system, the model pre-training system 120, the model fine-tuning system 130, and the model application system 140 can be deployed on any suitable electronic device. This electronic device can be any type of computing-capable device, including terminal devices or server devices. The terminal device can be any type of mobile terminal, fixed terminal, or portable terminal, including mobile phones, desktop computers, laptop computers, notebook computers, netbook computers, tablet computers, media computers, multimedia tablets, personal communication system (PCS) devices, personal navigation devices, personal digital assistants (PDAs), audio / video players, digital cameras / camcorders, positioning devices, television receivers, radio receivers, e-book devices, gaming devices, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof.

[0073] The server-side device can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and big data and artificial intelligence platforms. Server-side devices may include, for example, computing systems / servers, such as mainframes, edge computing nodes, computing devices in cloud environments, etc. It is understood that the model pre-training system 120, the model fine-tuning system 130, and the model application system 140 can be deployed on the same electronic device or on different electronic devices; this disclosure does not limit this.

[0074] It should be understood that Figure 1 The components and arrangements shown in the example environment 100 are merely examples, and a computing system suitable for implementing the exemplary implementations described in this disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown as separate, the model pre-training system 120, the model fine-tuning system 130, and the model application system 140 may be integrated in the same system or device. For example, at least the model pre-training system 120 and the model fine-tuning system 130 may be integrated in a model training system or device. Implementations of this disclosure are not limited in this respect.

[0075] Figure 2A An example architecture 200A for speech generation is shown. Architecture 200A can be implemented in model application system 140. Architecture 200A includes a scheduling module 210, multiple candidate second machine learning models 212, a first machine learning model 220, a verification module 230, and a decoding module 240.

[0076] The first machine learning model 220 may include a local encoding sub-model, an autoregressive sub-model, a residual acoustic sub-model, and a diffusion sub-model. The diffusion sub-model may, for example, be a flow matching architecture or a Transformer-based diffusion model. As an example, the weights of the first machine learning model 220 may remain constant during the training and application of the second machine learning model. The first machine learning model 220 can serve as an offline supervision source during the training phase and as a read-only intermediate feature provider, candidate verification model, and backtracking generation model during the application phase. The first machine learning model 220 may also be referred to as the target model.

[0077] The speech generation information 116 may include at least text 201 and timbre identifier 202. The model application system 140 may use the first machine learning model 220 to process the speech generation information 116 to obtain intermediate feature representation 222. The scheduling module 210 determines a target second machine learning model 214 that matches the speech generation information, based at least on the speech generation information 116. In some examples, the scheduling module 210 selects the target second machine learning model 214 that matches the speech generation information from multiple candidate second machine learning models 212, based at least on the speech generation information 116. The model size of the first machine learning model 220 is larger than the model size of any one of the multiple candidate second machine learning models 212. It should be noted that the two operations of the scheduling module 210 determining the target second machine learning model 214 and using the first machine learning model 220 to process the speech generation information 116 to obtain intermediate feature representation 222 can have any order, and they can also be executed in parallel.

[0078] In some cases, the model application system 140 can use the first machine learning model 220 to process the speech generation information 116, obtaining intermediate outputs from multiple intermediate layers. The model application system 140 can select at least one intermediate output from a specified layer and determine an intermediate feature representation 222 accordingly. The specified layer can be evenly distributed among the multiple intermediate layers of the first machine learning model 220. For example, the specified layer may include representative layers located at shallower, intermediate, and deeper positions. The layer positions, once determined during the model design phase, can remain consistent during the training and application of the second machine learning model.

[0079] The first machine learning model 220 may include an autoregressive sub-model and a diffusion sub-model. Multiple intermediate layers may include at least one layer from the autoregressive sub-model and at least one layer from the diffusion sub-model. Intermediate outputs from different sub-models may be normalized and linearly mapped before being fused into a context vector. The objective second machine learning model 214 may perform key-value injection in the self-attention processing of each layer. The model may concatenate the context vector with its own internal features along the sequence dimension, and then utilize shared key and value projections to generate the keys and values ​​of the current layer. This processing allows a smaller-scale model to acquire cross-layer information from the first machine learning model 220.

[0080] By extracting the output of a specified intermediate layer, the target second machine learning model 214 can obtain context related to the current speech generation conditions. This context reduces the amount of information that small-scale models need to learn independently. This allows for a reduction in the number of layers or hidden dimensions of the target second machine learning model 214 while maintaining generative coherence, and also reduces its computational cost.

[0081] Multiple candidate second machine learning models 212 can include first-type and second-type second machine learning models. The first-type second machine learning model can be considered a general-purpose model, and the second-type model a non-general-purpose model. The second-type second machine learning model can include third-type and fourth-type models. The third-type model can be considered a model for objects of a preset type. It is understood that the preset type of object can include multiple objects of the same type, but corresponding to different timbres. That is, the third-type model can target multiple timbres and generate audio for any of these timbres. The fourth-type model can be considered a model for a single timbre (i.e., the fourth-type model can only generate audio for the corresponding timbre).

[0082] The objective is that the second machine learning model 214 processes the speech generation information 116 based on the intermediate feature representation 222 and generates multiple speech feature representations 216. One of the speech feature representations 216 corresponds to an audio segment in the audio 118.

[0083] The verification module 230 can perform verification on multiple speech feature representations 216. At least one verified speech feature representation 235 is provided to the decoding module 240. The decoding module 240 generates at least one audio segment in the audio 118 based on the at least one speech feature representation 235. Each audio segment has a target timbre indicated by a timbre identifier 202.

[0084] Through architecture 200A, the speech generation task can obtain a matching target second machine learning model 214 from multiple candidate models. This model combines the intermediate feature representations 222 of the first machine learning model 220 to generate representations corresponding to audio segments in batches. This reduces the number of serial generation operations of the first machine learning model 220 and lowers the processing latency of the generated audio 118.

[0085] The objective of the second machine learning model 214 is to generate at least some of the speech feature representations in multiple speech feature representations 216 in parallel. The objective of the second machine learning model 214 can adopt a block-level diffusion structure or a block-level flow matching structure. Information can be passed bidirectionally between positions within a block, information can be passed unidirectionally from a block to the context, and different blocks can be isolated from each other.

[0086] In some cases, the target second machine learning model 214 directly outputs N final audio latent vectors through a single forward processing and K-step ordinary differential equation solving. K can be less than the number of solution steps M used by the diffusion sub-model of the first machine learning model 220. This forward processing can handle the four stages of local encoding, autoregressive processing, residual acoustic modeling, and diffusion decoding without requiring secondary processing, completion, or correction from any layer in the first machine learning model 220. For example, K can be 3 to 5, and M can be 10. N can be chosen from 4, 6, 8, and 16, or other positive integers. N can be dynamically adjusted during application based on computational resources, reception length, or latency requirements without retraining the target second machine learning model 214. By replacing the four stages of processing, the number of calls to both the autoregressive and diffusion sub-models in the first machine learning model 220 decreases with the average length of the received prefixes. Compared to processing that only reduces the number of calls to the autoregressive sub-model, the diffusion sub-model no longer performs M-step solutions for each candidate position. This can further reduce serial model calls and reduce the computational resources required for multi-step denoising.

[0087] Each speech feature representation 216 can be a floating-point vector. A floating-point vector can correspond to a continuous audio latent vector output by an audio encoder, or it can correspond to an audio block containing multiple consecutive frames. Of course, in some cases, each speech feature representation 216 can also be a discrete feature representation; this is not a limitation.

[0088] The second objective, machine learning model 214, can generate multiple candidate audio segments in parallel, enabling a single model call to propose multiple audio segments. Continuous vectors can preserve fine-grained acoustic information in the compressed audio representation. This combination reduces position-wise model calls and avoids the additional computation of converting continuous representations into discrete indices.

[0089] In some cases, the verification module 230 verifies each speech feature representation 216 sequentially according to the order of these representations. This order corresponds to the order of the audio segments associated with each speech feature representation in the audio 118. The decoding module 240 uses the verified speech feature representations 235 to generate the corresponding audio segments.

[0090] Verification can employ a prefix acceptance method. Verification module 230 begins processing from the first speech feature representation. If multiple consecutive speech feature representations pass verification, verification module 230 retains that consecutive prefix. If the first speech feature representation fails verification, subsequent candidate representations at that position can be discarded.

[0091] Sequential verification ensures that accepted audio segments remain temporally continuous. Continuous prefixes can directly enter the decoding module 240 without waiting for all candidate positions to be generated serially. This process reduces the number of duplicate verifications and decodings.

[0092] For the first speech feature representation, the verification module 230 can determine a first log probability density and a second log probability density along the denoising trajectory using the first machine learning model 220 and the target second machine learning model 214, respectively. The two log probability densities indicate the credibility of the first speech feature representation under the corresponding model distributions. The verification module 230 calculates the acceptance probability based on the difference between the two, and determines that the representation passes verification when the acceptance probability reaches a second threshold.

[0093] In one implementation, the acceptance probability represents the i-th speech feature. It can be represented as .in This represents the first logarithmic probability density determined by the first machine learning model 220. This represents the second logarithmic probability density determined by the second machine learning model 214. The function min represents taking the smaller of the two inputs, and the function exp represents the exponential operation.

[0094] Validation based on probability density relationships allows acceptance criteria to be correlated with the credibility of the two models for the same candidate representation. This approach provides a computable basis for accepting speech segments. When the acceptance criteria are met, the system can reuse candidate results and reduce the number of times the first machine learning model 220 performs full generation.

[0095] In the case of using probability density ratios, the verification module 230 can utilize the instantaneous variable substitution relationship in the flow matching process and estimate the divergence along the ordinary differential equation trajectory. The acceptance probability can be the smaller of 1 and the exponential value of the difference between the first and second log probability densities. Performing rejection sampling according to this acceptance probability can make the validated samples conform to the target distribution corresponding to the first machine learning model 220.

[0096] If the second speech feature representation fails validation, the model application system 140 can use the first machine learning model 220 to generate one or more speech feature representations 224. This generation is based on a portion of the speech feature representations that have previously been validated by the second speech feature representation. One or more speech feature representations 224 replace the second speech feature representation and the subsequent candidate representations.

[0097] The decoding module 240 can generate one or more audio segments based on one or more speech feature representations 224. The final audio 118 includes audio segments generated based on validated representations and audio segments generated backward based on the first machine learning model 220. If all candidate representations fail validation, the first machine learning model 220 can continue to perform complete generation from the current generation position.

[0098] The rollback process preserves accepted prefixes up to the position that failed verification and regenerates from the first position that failed verification. This process ensures that the speech generation process can continue. By avoiding recalculation of accepted prefixes, the model application system 140 reduces redundant inference overhead and cache writes.

[0099] The validation module 230 can select one or a combination of various validation strategies. Different strategies can be switched during inference without requiring retraining of the target second machine learning model 214. The following table lists examples of validation strategies.

[0100] Table 1

[0101] The sample distance can be represented by cosine distance. Target velocity field scoring can be performed when denoising is nearly complete. The hidden state space distance can be provided by the auxiliary prediction head of the second machine learning model 214 for the target. The flow matching probability density ratio can be determined by combining the divergence estimate along the ordinary differential equation trajectory.

[0102] The acceptance thresholds used for each validation strategy can be calibrated using a small validation dataset. The model application system 140 can switch validation strategies or thresholds during application based on audio quality and latency requirements. This switching does not change the trained parameters of the target second machine learning model 214.

[0103] The first machine learning model 220 can perform parallel verification of N candidate speech feature representations in a single batch computation. The verification module 230 then makes prefix acceptance decisions according to the order of audio segments. Therefore, parallel model computation and sequential determination of acceptance prefixes can be implemented simultaneously without contradiction.

[0104] For consecutive prefixes that pass validation, the model application system 140 can push the corresponding key-value data to the autoregressive cache. If a candidate representation fails validation, the model application system 140 can roll back the residual cache corresponding to that position and subsequent candidate positions. The model application system 140 then uses the first machine learning model 220 to generate additional speech feature representations for the positions that failed validation, and uses these representations as the context for generating the next batch of candidates.

[0105] The model application system 140 can repeatedly perform candidate generation, verification, and decoding. The stop prediction module can determine whether the audio corresponding to the text has been generated completely based on the current speech feature representation, the generated content, or the stop marker. If it has not been completed, the model application system 140 continues to generate the next batch of speech feature representations. If it has been completed, the decoding module 240 outputs audio 118.

[0106] Multiple candidate second machine learning models 212 can be trained separately. Before training, the system can run the first machine learning model 220 on text-audio pairing data and cache the training record consisting of text, intermediate feature representations, and audio latent vectors. When training the second machine learning model, the cached record can be read directly to reduce repeated calls to the first machine learning model 220.

[0107] The second machine learning model of the first type can be trained using a block-level flow matching loss. This loss can be a position-weighted L2 distance between the candidate velocity field prediction and the ground truth values ​​of the noise and audio latent vectors. During training, conditional branches can be dropped with a probability of 0.1, for example, to support classifier-free guidance. This probability can also be adjusted based on the training data and generation quality requirements.

[0108] Training can also employ at least one of the following: stopping prediction head loss, diffuse submodel hidden state consistency loss, and density ratio head loss. The stopping prediction head loss is used to learn when to output the stopping marker. The hidden state consistency loss can support hidden state spatial distance verification, and the density ratio head loss can support probability density ratio verification.

[0109] During the pedestal training phase, target timbre identity information can be discarded, or data from multiple target timbres can be used in the same batch. This processing enables the second machine learning models of types one and two to learn prosodic structures such as speech rate, pauses, and intonation, independent of specific timbres. In some training configurations, this structure can reduce the number of fine-tuning steps for type four models by two to three orders of magnitude.

[0110] The first type of second machine learning model can be trained using data covering multiple timbres. The third type of second machine learning model can incorporate attribute information during training. Attribute information indicates the timbre style associated with objects of a preset type, with different attribute information corresponding to different preset types. The fourth type of second machine learning model can be trained using samples corresponding to a specified timbre. The first machine learning model 220 can keep its parameters unchanged and serve as a read-only intermediate feature provider.

[0111] In some cases, a second machine learning model of type four can be obtained through multiple paths. These paths can be used individually or in combination, provided the material explicitly allows it. The table below lists examples of suitable paths.

[0112] Table 2

[0113] The fourth type of model can further accumulate and inject target timbre identifiers, reference audio representations, and fine-grained attributes based on the prior information of the third type of model. Fine-grained attributes may include at least one of speech rate, average fundamental frequency, and energy envelope. These continuous attributes, along with attribute information used to define a wider set of timbres, can be encoded separately and concatenated to the beginning of the key-value injection sequence.

[0114] The forward interface of the first machine learning model 220 may include a Boolean parameter indicating whether to output hidden states, and a list of layer indices indicating the layers to be output. The first machine learning model 220 appends the hidden states corresponding to the returned layer index list only when the Boolean parameter is enabled. This interface extension does not change the weights of the first machine learning model 220, nor does it change the output of the default inference path.

[0115] For example, a specified layer may include layer 2, layer 12, and layer 23 to provide information from shallower, intermediate, and deeper layers, respectively. The outputs of each layer can be fused after root mean square normalization and linear mapping. The specific layer number can be adjusted according to the number of layers in the first machine learning model 220.

[0116] These adaptation paths allow candidate models to be obtained without updating the parameters of the first machine learning model. Reusing training records reduces the steps required for data preparation. Smaller, specialized models also reduce runtime storage and computational costs.

[0117] Figure 2B An example process 200B of scheduling module 210 scheduling a second machine learning model is shown. Process 200B can be executed by model application system 140. Scheduling module 210 can select target second machine learning model 214 by combining speech generation information 116, application frequency, and model state.

[0118] In some cases, the scheduling module 210 selects a target second machine learning model 214 from multiple candidate second machine learning models 212 based on the speech generation information 116 and the application frequency of the target timbre within a predetermined time period. The application frequency can be the number of calls within a sliding time window or the ratio of the number of calls to the window duration.

[0119] The selection based on application frequency allows the model application system 140 to use different candidate models for speech generation tasks at different frequencies. High-frequency tasks can obtain models with a narrower applicability, while low-frequency tasks can share models with a wider applicability. This reduces the resident storage overhead of dedicated models while maintaining the availability of processing paths.

[0120] In box 251, the scheduling module 210 determines whether the speech generation information 116 includes attribute information. Attribute information indicates the timbre style associated with an object of a preset type. Attribute information may indicate language, gender, age, etc. The scheduling module 210 can read the attribute fields in the speech generation information 116 and determine whether the field has a valid value.

[0121] If the speech generation information 116 does not include attribute information, then the scheduling module 210 executes box 252. In box 252, the scheduling module 210 selects the second machine learning model of the first type as the target second machine learning model 214.

[0122] If the speech generation information 116 includes attribute information, then the scheduling module 210 executes box 253. In box 253, the scheduling module 210 selects a target second machine learning model 214 from the second type of second machine learning models based on the application frequency.

[0123] The first type of second machine learning model corresponds to the first timbre set. The second type of second machine learning model corresponds to the second timbre set, which is a subset of the first timbre set. This branch allows tasks lacking attribute information to still use the shared model, while allowing tasks with attribute information to enter a narrower set of candidates. This reduces the number of calls and loading of mismatched candidate models.

[0124] In box 254, scheduling module 210 determines whether the application frequency of the target timbre reaches a first threshold. If the application frequency does not reach the first threshold, scheduling module 210 executes box 255. In box 255, scheduling module 210 can select a third type of second machine learning model as the target second machine learning model 214. If the application frequency reaches the first threshold, scheduling module 210 executes box 256.

[0125] In box 256, scheduling module 210 can use the fourth type of second machine learning model as the target second machine learning model 214. In this case, scheduling module 210 can execute box 257. In box 257, scheduling module 210 can determine whether the fourth type of second machine learning model includes a second machine learning model for the target timbre. If the fourth type of second machine learning model includes a model for the target timbre, scheduling module 210 executes box 258. In box 258, scheduling module 210 can also determine whether the second machine learning model for the target timbre in the fourth type of second machine learning model is available. If the second machine learning model for the target timbre is available, scheduling module 210 executes box 259. In box 259, scheduling module 210 can select the second machine learning model corresponding to the target timbre in the fourth type of second machine learning model.

[0126] In some examples, if the second machine learning model of type four does not include a model for the target timbre, the scheduling module 210 executes box 255 to select a second machine learning model of type three. In some examples, if the second machine learning model for the target timbre is unavailable, the scheduling module 210 executes box 255 to select a second machine learning model of type three. Thus, by combining application frequency and availability, the selection allows the current task to continue using a multi-timbre model when a dedicated model is unavailable. This fallback path avoids waiting for unavailable models, thereby reducing request blocking and invalid model loading. The two-level determination of model existence and availability allows the scheduling module 210 to prioritize the use of ready single-timbre models and provide fallback models when such models are missing or unavailable. This maintains the continuous availability of the speech generation path. The system also avoids repeatedly attempting to call unready models, thereby reducing failed calls and status lookup overhead.

[0127] In some examples, if the second machine learning model for the target timbre is unavailable or the fourth type of second machine learning model does not include a model for the target timbre, the scheduling module 210 can also execute box 260. In box 260, the scheduling module 210 can trigger training of the second machine learning model for the target timbre. In this case, the model application system 140 can train the second machine learning model for the target timbre. After training, the model application system 140 switches the model from an unavailable state to an available state. The training task can be executed asynchronously in background computing resources. During training, online speech generation continues to use the first, second, or third type of second machine learning model. After training, the scheduling module 210 can atomically update the model state and routing relationships. Thus, if the second machine learning model needs to be called again later, the scheduling module 210 can select the second machine learning model for the target timbre.

[0128] On-demand training allows for the development of dedicated candidate models for target timbres as their application frequency increases, without requiring a pause in the current speech generation task. Background execution reduces the impact of training on foreground responses. State switching also prevents incomplete models from participating in inference, thereby reducing the probability of erroneous calls and repeated rollbacks.

[0129] The fourth type of second machine learning model may include a third machine learning model for the first timbre. When the third machine learning model is available, if the application frequency of the first timbre does not reach the third threshold, the model application system 140 can switch the third machine learning model to an unavailable state. The third machine learning model remains unavailable for a preset duration.

[0130] The preset duration can serve as a cooldown period. Before the cooldown period ends, even if the application frequency increases again, the scheduling module 210 can keep the third machine learning model in an unavailable state. This process reduces the repeated switching of model states in a short period of time. The number of model loading, unloading, and training triggers is reduced accordingly, thereby reducing memory jitter and computational overhead.

[0131] In some cases, the second type of second machine learning model is obtained by fine-tuning the first type of second machine learning model. The fourth type of second machine learning model is obtained by fine-tuning the third type of second machine learning model. By fine-tuning based on existing candidate models, model parameters and training records can be reused. This approach shortens the preparation path to obtain subdivided models or monotonic models and reduces the amount of computation and training data required for training from initial parameters.

[0132] Alternatively, the second type of second machine learning model can be obtained by distilling the first type of second machine learning model, or it can be trained independently using overlapping training data. The fourth type of second machine learning model can also be obtained by employing at least one of the following paths: low-rank adaptation, structured pruning, knowledge distillation, subnet activation, or independent small model training.

[0133] The scheduling module 210 can also set an increase threshold and a decrease threshold. The two thresholds form a hysteresis interval to reduce state jitter near the critical frequency. The interval between the increase threshold and the decrease threshold can, for example, be a ratio of 5 or more.

[0134] The model application system 140 can limit the number of Type IV second machine learning models that can be available simultaneously. If the limit is reached, the system can switch less frequently used models to an unavailable state according to the Least Recently Used (LRU) rule. The task corresponding to the switched model can be rolled back to a Type II or Type III second machine learning model based on attribute information.

[0135] Resource caps and the Least Recently Used (LRU) rule prioritize allocating limited storage space to more frequently used monophonic models. A fallback path ensures that tasks corresponding to unloaded models can still access shared candidate models. This approach controls the overall storage footprint of dedicated models and improves the effective utilization of model caches.

[0136] In scenarios where a fixed target timbre is frequently used, the scheduling module 210 can select a second machine learning model of the fourth type. In scenarios where the attribute information is known but the target timbre is used less frequently, the scheduling module 210 can select a second machine learning model of the third type. In scenarios where the attribute information is unknown, the scheduling module 210 can select a second machine learning model of the first type. In dynamically switching scenarios, the model can migrate between different types as the application frequency and attribute information change.

[0137] In some examples, the scheduling module 210 can maintain a first-type state, a third-type state, a fourth-type training state, and a fourth-type available state for a target timbre. Upon receiving input, if the attribute information is unknown, the scheduling module 210 moves it to the first-type state. If the attribute information is known, the scheduling module 210 moves it to the third-type state. Of course, if the attribute information changes from unknown to known (e.g., the user manually inputs attribute information), the scheduling module 210 can migrate the target timbre from the first-type state to the third-type state. If the attribute information becomes invalid, the scheduling module 210 can migrate the target timbre back to the first-type state. Both the first-type and third-type states can migrate to the fourth-type training state at the end of the application frequency reaching the promotion threshold and the cooling-off period (i.e., the period when the model is in an unavailable state).

[0138] After the fourth type of model training is completed, the scheduling module 210 can atomically transition the target timbre from the state in the fourth type training to the fourth type available state. If an anomaly occurs during training, the scheduling module 210 can revert the target timbre to the first type state. Online requests during training continue to use the first, second, or third type of model.

[0139] If the application frequency falls below the degradation threshold, or if the fourth type model is unloaded according to the least recently used rule, the scheduling module 210 causes the target timbre to exit the fourth type available state. When the attribute information is valid, the target timbre can migrate to the third type state. When the attribute information is unavailable, the target timbre can migrate to the first type state.

[0140] The predetermined time period used for statistical application frequency can be a sliding time window, such as 24 hours. The promotion threshold can be higher than the demotion threshold, forming a hysteresis interval. After a target timbre is degraded from the fourth type of available state, it can be allowed to be promoted again only after a cooling-off period of, for example, a predetermined duration (e.g., 7 days) (during which the model remains in an unavailable state).

[0141] Thus, the model application system 140 can provide continuously available candidate model paths for different speech generation conditions. The multi-level candidate model avoids having a separate model for all target timbres. This architecture strikes a balance between inference acceleration for frequently used tasks and storage costs for less frequently used tasks, and improves concurrent speech generation capabilities.

[0142] The candidate generation and verification processes described above can also be used in other generation systems with autoregressive processing, diffusion processing, or stream matching processing, as well as continuous latent representations. For example, this processing can be used for image generation, video generation, or multimodal content generation with continuous representations. In these systems, the speech feature representation can be replaced with continuous feature blocks of images, videos, or multimodal content, respectively.

[0143] The implementation of Architecture 200A and Process 200B can also be validated through running features. For example, the number of calls to the autoregressive and diffusion sub-models in the first machine learning model can be compared with the total number of output feature blocks. When replacing all four segments, the number of calls to both types of sub-models can be lower than the total number of output feature blocks, and is related to the average length of the accepted prefixes. It can also be analyzed whether the model file contains both the first machine learning model and a significantly smaller second machine learning model, and whether the second machine learning model includes cross-layer hidden state projection parameters. For a target timbre-specific model, it can be analyzed whether adding a new target timbre results in a small-scale adaptation weight, while the weights of the first machine learning model remain unchanged. For dynamic scheduling, it can be compared whether the end-to-end latency and model weight occupancy in different application frequency ranges exhibit a stepwise change corresponding to the candidate model level.

[0144] Figure 3 A flowchart of an example speech generation process 300 based on certain scenarios is shown. Process 300 can be implemented in the model application system 140. See below for reference. Figure 3 To describe process 300.

[0145] like Figure 3 As shown in box 310, the model application system 140 can perform the following operations: process speech generation information using a first machine learning model to obtain intermediate feature representations, wherein the speech generation information includes text and timbre identifiers; In box 320, the model application system 140 may perform the following operations: at least based on the speech generation information, determine a target second machine learning model that matches the speech generation information, wherein the model size of the first machine learning model is larger than that of the second machine learning model. In box 330, model application system 140 can perform the following operations: utilize the target second machine learning model to process speech generation information based on intermediate feature representations to generate multiple speech feature representations, the multiple speech feature representations corresponding to audio for text, one of which corresponds to an audio segment in the audio; and In box 340, the model application system 140 may perform the following operations: generate at least one audio segment in an audio file based on at least one of a plurality of speech feature representations, the audio having a target timbre indicated by a timbre identifier.

[0146] Through process 300, the speech generation task can select an applicable model among multiple candidate models and use that model to generate representations of audio segments in batches. This process reduces the number of serial generation iterations of the first machine learning model. This shortens the audio output latency and improves the inference throughput of the model application system 140.

[0147] In some examples, the model application system 140 may perform the following operations: at least based on speech generation information, select a target second machine learning model that matches the speech generation information from multiple candidate second machine learning models, wherein the model size of the first machine learning model is larger than the model size of the candidate second machine learning models.

[0148] In some scenarios, the model application system 140 may perform the following operations: selecting a target second machine learning model that matches the speech generation information from multiple candidate second machine learning models, based at least on the speech generation information; including selecting a target second machine learning model from multiple candidate second machine learning models based on the application frequency of the speech generation information and the target timbre within a predetermined time period. This process enables speech generation tasks with different application frequencies to obtain matching candidate models. This reduces unnecessary dedicated model residency and lowers storage requirements.

[0149] In some scenarios, the model application system 140 may perform the following operations: selecting a target second machine learning model that matches the speech generation information from multiple candidate second machine learning models, based at least on the speech generation information: determining whether the speech generation information includes attribute information, where the attribute information indicates the timbre style associated with an object of a preset type, and different attribute information corresponds to different preset types; in response to the speech generation information not including attribute information, selecting a second machine learning model of a first type from multiple candidate second machine learning models as the target second machine learning model; or in response to the speech generation information including attribute information, selecting a second machine learning model of a second type from multiple candidate second machine learning models as the target second machine learning model based on application frequency, wherein the second machine learning model of the first type corresponds to a first timbre set, the second machine learning model of the second type corresponds to a second timbre set, and the second timbre set is a subset of the first timbre set. This process simultaneously covers input conditions where the attributes are known and unknown. This can reduce the size of the candidate set and lower the inference overhead of mismatched models.

[0150] In some scenarios, the model application system 140 may perform the following operations: selecting a second type of second machine learning model from multiple candidate second machine learning models based on application frequency, including: selecting a third type or fourth type of second machine learning model from multiple candidate second machine learning models as the target second machine learning model based on application frequency and whether the second type of second machine learning model is available, wherein the third type of second machine learning model corresponds to multiple timbres, and the fourth type of second machine learning model corresponds to a single timbre. This process provides a shared model path when a dedicated model is unavailable. This reduces request blocking and invalid model loading.

[0151] In some scenarios, the model application system 140 can perform the following operations: selecting a third-type or fourth-type second-machine learning model from multiple candidate second-machine learning models as the target second-machine learning model. This includes: in response to the application frequency reaching a first threshold, selecting a second-machine learning model from the third-type or fourth-type second-machine learning models as the target second-machine learning model based on whether the second-type second-machine learning models are available; or in response to the application frequency not reaching the first threshold, selecting a third-type second-machine learning model from the second-type second-machine learning models as the target second-machine learning model. This process adapts model selection to task frequency. This allows control over the number of dedicated models and resident memory.

[0152] In some scenarios, the model application system 140 may perform the following operations: Selecting a second machine learning model from either a third-type or a fourth-type second machine learning model as the target second machine learning model includes: in response to the fourth-type second machine learning model including a second machine learning model for the target timbre, selecting the second machine learning model for the target timbre as the target second machine learning model based on its availability; and selecting a third-type second machine learning model as the target second machine learning model based on its unavailability. This process prioritizes the use of available single-timbre models and falls back when unavailable. This reduces the overhead of failed calls and status queries.

[0153] In some cases, the model application system 140 may perform the following operations: the method further includes: training a second machine learning model for the target timbre in response to the fourth type of second machine learning model not including a second machine learning model for the target timbre or the second machine learning model for the target timbre being unavailable; and switching the second machine learning model for the target timbre from an unavailable state to an available state in response to the completion of training of the second machine learning model for the target timbre. This process supplements candidate models without interrupting current speech generation. Background training can reduce foreground response latency.

[0154] In some scenarios, the model application system 140 may perform the following operations: The second machine learning model of the fourth type includes a third machine learning model, which is applied to the first timbre. The method further includes: in response to determining that the third machine learning model is available and the application frequency of the first timbre has not reached a third threshold, switching the third machine learning model from an available state to an unavailable state, wherein the third machine learning model remains unavailable for a preset duration. This processing reduces model switching caused by short-term frequency fluctuations. This reduces the number of model loading and unloading operations.

[0155] In some cases, the second type of second machine learning model is obtained by fine-tuning the first type of second machine learning model, and the fourth type of second machine learning model is obtained by fine-tuning the third type of second machine learning model. This process shortens the path to obtaining the subdivided models. This allows for the reuse of model parameters and reduces training computation.

[0156] In some scenarios, the model application system 140 may perform the following operations: processing speech generation information using a first machine learning model to obtain intermediate feature representations includes: processing the speech generation information using the first machine learning model to obtain intermediate outputs from multiple intermediate layers of the first machine learning model; selecting at least one intermediate output from a specified layer from the intermediate outputs of the multiple intermediate layers; and determining intermediate feature representations based on the intermediate outputs of at least one specified layer. This processing provides cross-layer context for the target second machine learning model. This can reduce the computational overhead of independently extracting context for small-scale models.

[0157] In some cases, at least one designated layer is evenly distributed across multiple intermediate layers of the first machine learning model. This setup allows intermediate feature representations to simultaneously encompass model information at different depths. This reduces information redundancy and cache usage caused by centrally reading adjacent layers.

[0158] In some cases, the first machine learning model includes an autoregressive sub-model and a diffusion sub-model, with multiple intermediate layers including at least one layer from the autoregressive sub-model and at least one layer from the diffusion sub-model. This setup allows the target second machine learning model to simultaneously acquire linguistic context and acoustic generation information. This can reduce candidate representation bias and the resulting backoff computations.

[0159] In some cases, the objective-second machine learning model is configured to generate at least a portion of the speech feature representations from multiple speech feature representations in parallel. This process allows a single model call to produce multiple candidate audio segments. This reduces the number of serial calls and improves inference throughput.

[0160] In some cases, each speech feature in a multi-speech feature representation is represented as a floating-point vector. Floating-point vectors can continuously represent the acoustic information of an audio segment. This avoids discretization and dediscretization processes and reduces additional transformation computations.

[0161] In some cases, the model application system 140 may perform the following operations: It further includes: sequentially verifying each of the multiple speech feature representations in the order of the multiple speech feature representations, the order of which corresponds to the order of the audio segments corresponding to each of the multiple speech feature representations in the audio; wherein at least one speech feature representation used to generate at least one audio segment is a verified speech feature representation among the multiple speech feature representations. This process maintains the temporal continuity of the accepted audio prefixes. This reduces redundant verification and decoding.

[0162] In some cases, the model application system 140 can perform the following operations: Validating each speech feature representation among multiple speech feature representations includes: for a first speech feature representation among the multiple speech feature representations, determining a first logarithmic probability density and a second logarithmic probability density of the first speech feature representation along a denoising trajectory using a first machine learning model and a target second machine learning model, respectively, where the first and second logarithmic probability densities indicate the credibility of the first speech feature representation; calculating the acceptance probability of the first speech feature representation based on the difference between the first and second logarithmic probability densities; and determining whether the first speech feature representation passes validation based on the acceptance probability, wherein if the acceptance probability reaches a second threshold, the first speech feature representation is determined to have passed validation. This process provides acceptance criteria related to the distributions of the two models. This allows for the reuse of candidate representations that meet the conditions and reduces the number of complete generation iterations.

[0163] In some cases, the model application system 140 may perform the following operations: In response to a second speech feature representation among multiple speech feature representations failing validation, using a first machine learning model, one or more speech feature representations are generated based on a subset of speech feature representations that have passed validation prior to the second speech feature representation among the multiple speech feature representations. These one or more speech feature representations replace the second speech feature representation and subsequent speech feature representations among the multiple speech feature representations; and one or more audio segments are generated based on the one or more speech feature representations, the audio including at least one audio feature segment and one or more audio feature segments. This process preserves accepted prefixes and continues generation from the failed position. This avoids recalculating accepted content and reduces inference and caching overhead.

[0164] A corresponding apparatus for implementing the above methods or processes is also provided.

[0165] Figure 4 Block diagrams of a speech generation device 400 in several scenarios are shown. The device 400 can be implemented as a model application system 140 or included in an electronic device 500. The various modules in the device 400 can be implemented by hardware, software, firmware, or any combination thereof.

[0166] like Figure 4As shown, the apparatus 400 includes a processing module 410, a determining module 420, a generating module 430, and an audio output module 440. The processing module 410 is configured to perform the following operations: process speech generation information using a first machine learning model to obtain intermediate feature representations, wherein the speech generation information includes text and timbre identifiers; the determining module 420 is configured to perform the following operations: determine a target second machine learning model that matches the speech generation information, at least based on the speech generation information, wherein the model size of the first machine learning model is larger than that of the second machine learning model; the generating module 430 is configured to perform the following operations: process the speech generation information based on the intermediate feature representations using the target second machine learning model to generate multiple speech feature representations, the multiple speech feature representations corresponding to audio for text, one of which corresponds to an audio segment in the audio; and the audio output module 440 is configured to perform the following operations: generate at least one audio segment in the audio, the audio having a target timbre indicated by a timbre identifier, based on at least one of the multiple speech feature representations.

[0167] In some cases, the determining module 420 may also be configured to perform the following operations: at least based on the speech generation information, select a target second machine learning model that matches the speech generation information from a plurality of candidate second machine learning models, wherein the model size of the first machine learning model is larger than the model size of the candidate second machine learning models.

[0168] In some cases, the determining module 420 may also be configured to perform the following operations: at least based on the speech generation information, selecting a target second machine learning model that matches the speech generation information from multiple candidate second machine learning models, including: selecting a target second machine learning model from multiple candidate second machine learning models based on the speech generation information and the application frequency of the target timbre within a predetermined time period.

[0169] In some cases, the determining module 420 may also be configured to perform the following operations: at least based on the speech generation information, selecting a target second machine learning model that matches the speech generation information from a plurality of candidate second machine learning models, including: determining whether the speech generation information includes attribute information, the attribute information indicating the timbre style associated with an object of a preset type, different attribute information corresponding to different preset types; in response to the speech generation information not including attribute information, selecting a second machine learning model of a first type from a plurality of candidate second machine learning models as the target second machine learning model; or in response to the speech generation information including attribute information, selecting a second machine learning model of a second type from a plurality of candidate second machine learning models as the target second machine learning model based on the application frequency, wherein the second machine learning model of the first type corresponds to a first timbre set, the second machine learning model of the second type corresponds to a second timbre set, and the second timbre set is a subset of the first timbre set.

[0170] In some cases, the determining module 420 may also be configured to perform the following operations: selecting a second type of second machine learning model from multiple candidate second machine learning models based on the application frequency, including: selecting a third type of second machine learning model or a fourth type of second machine learning model from multiple candidate second machine learning models as the target second machine learning model based on the application frequency and whether the second type of second machine learning model is available, wherein the third type of second machine learning model corresponds to multiple timbres and the fourth type of second machine learning model corresponds to a single timbre.

[0171] In some cases, the determining module 420 may also be configured to perform the following operations: selecting a third type of second machine learning model or a fourth type of second machine learning model from a plurality of candidate second machine learning models as the target second machine learning model, including: in response to the application frequency reaching a first threshold, selecting a second machine learning model from the third type of second machine learning model or the fourth type of second machine learning model as the target second machine learning model based on whether the second type of second machine learning model is available; or in response to the application frequency not reaching the first threshold, selecting a third type of second machine learning model from the second type of second machine learning models as the target second machine learning model.

[0172] In some cases, the determining module 420 may also be configured to perform the following operations: wherein selecting a second machine learning model from a third type of second machine learning model or a fourth type of second machine learning model as the target second machine learning model includes: in response to the fourth type of second machine learning model including a second machine learning model for the target timbre, selecting a second machine learning model for the target timbre as the target second machine learning model based on the second machine learning model for the target timbre being available; and selecting a third type of second machine learning model as the target second machine learning model based on the second machine learning model for the target timbre being unavailable.

[0173] In some cases, the determination module 420 may also be configured to perform the following operations: the method further includes: training a second machine learning model for the target timbre in response to the second machine learning model of the fourth type not including a second machine learning model for the target timbre or the second machine learning model for the target timbre being unavailable; and switching the second machine learning model for the target timbre from an unavailable state to an available state in response to the completion of training of the second machine learning model for the target timbre.

[0174] In some cases, the determination module 420 may also be configured to perform the following operations: the second machine learning model of the fourth type includes a third machine learning model for the first timbre, and the method further includes: in response to determining that the third machine learning model is in an available state and the application frequency of the first timbre has not reached a third threshold, switching the third machine learning model from an available state to an unavailable state, wherein the third machine learning model remains in an unavailable state for a preset duration.

[0175] In some cases, the second type of second machine learning model is obtained by fine-tuning the first type of second machine learning model, and the fourth type of second machine learning model is obtained by fine-tuning the third type of second machine learning model.

[0176] In some cases, the processing module 410 may also be configured to perform the following operations: processing speech generation information using a first machine learning model to obtain intermediate feature representations, including: processing the speech generation information using the first machine learning model to obtain intermediate outputs of multiple intermediate layers of the first machine learning model; selecting at least one intermediate output of a specified layer from the intermediate outputs of the multiple intermediate layers; and determining intermediate feature representations based on the intermediate outputs of at least one specified layer.

[0177] In some cases, at least one designated layer is uniformly distributed among multiple intermediate layers of the first machine learning model.

[0178] In some cases, the first machine learning model includes an autoregressive sub-model and a diffusion sub-model, and multiple intermediate layers include at least one layer in the autoregressive sub-model and at least one layer in the diffusion sub-model.

[0179] In some cases, the objective-second machine learning model is configured to generate at least a portion of the speech feature representations from multiple speech feature representations in parallel.

[0180] In some cases, each speech feature in multiple speech feature representations is represented as a floating-point vector.

[0181] In some cases, the processing module 410 and the audio output module 440 may also be configured to perform the following operations in concert: further comprising: sequentially verifying each of the multiple speech feature representations in the order of the multiple speech feature representations, the order of the multiple speech feature representations corresponding to the order of the audio segments corresponding to each of the multiple speech feature representations in the audio; wherein at least one speech feature representation used to generate at least one audio segment is a speech feature representation that has passed verification among the multiple speech feature representations.

[0182] In some cases, the processing module 410 may also be configured to perform the following operations: verifying each speech feature representation among a plurality of speech feature representations includes: for a first speech feature representation among the plurality of speech feature representations, determining a first logarithmic probability density and a second logarithmic probability density of the first speech feature representation along a denoising trajectory using a first machine learning model and a target second machine learning model, respectively, wherein the first logarithmic probability density and the second logarithmic probability density indicate the credibility of the first speech feature representation; calculating the acceptance probability of the first speech feature representation based on the difference between the first logarithmic probability density and the second logarithmic probability density; and determining whether the first speech feature representation passes verification based on the acceptance probability, wherein if the acceptance probability reaches a second threshold, the first speech feature representation is determined to pass verification.

[0183] In some cases, the generation module 430 and the audio output module 440 may also be configured to perform the following operations in concert: further comprising: in response to a second speech feature representation among a plurality of speech feature representations failing validation, generating one or more speech feature representations based on a portion of the speech feature representations among the plurality of speech feature representations that have passed validation prior to the second speech feature representation, using a first machine learning model, the one or more speech feature representations being used to replace the second speech feature representation and the speech feature representations following the second speech feature representation among the plurality of speech feature representations; and generating one or more audio segments based on the one or more speech feature representations, the audio including at least one audio feature segment and one or more audio feature segments.

[0184] The modules included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some cases, one or more modules can be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the modules in device 400 can be implemented at least partially by one or more hardware logic components. By way of example and not limitation, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems on a chip (SOCs), or complex programmable logic devices (CPLDs) can be used.

[0185] Figure 5 A block diagram of an electronic device 500 in which one or more examples may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the examples described herein. Figure 5 The electronic device 500 shown can be used to implement the model application system 140 or device 400 as discussed above.

[0186] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processing units or processors 510, memory 520, storage devices 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processor 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processors execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0187] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof). Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0188] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various examples.

[0189] The communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 500 can operate in a networked environment using logical connections to one or more other servers, networked personal computers, or another network node.

[0190] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices, such as storage devices, display devices, etc., via communication unit 540 as needed, or with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via an input / output (I / O) interface (not shown).

[0191] A computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. A computer program product may also be provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0192] The flowcharts and / or block diagrams of the methods, apparatus, devices, and computer program products referred to herein describe various aspects. It should be understood that each block of the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0193] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0194] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0195] The flowcharts and block diagrams in the accompanying figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products under various scenarios. In this respect, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0196] Various examples have been described above. The foregoing descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A speech generation method, characterized in that, include: The speech generation information is processed using a first machine learning model to obtain intermediate feature representations, wherein the speech generation information includes text and timbre identifiers; Based at least on the speech generation information, a target second machine learning model that matches the speech generation information is determined, wherein the model size of the first machine learning model is larger than the model size of the target second machine learning model; The target second machine learning model processes the speech generation information based on the intermediate feature representations to generate multiple speech feature representations, each corresponding to audio for the text, wherein one speech feature representation corresponds to an audio segment in the audio; and Based on at least one of the plurality of speech feature representations, at least one audio segment of the audio is generated, the audio having the target timbre indicated by the timbre identifier.

2. The method according to claim 1, characterized in that, The step of determining the target second machine learning model that matches the speech generation information based at least on the speech generation information includes: Based at least on the speech generation information, a target second machine learning model that matches the speech generation information is selected from multiple candidate second machine learning models, wherein the model size of the first machine learning model is larger than the model size of the candidate second machine learning models.

3. The method according to claim 2, characterized in that, Based at least on the speech generation information, selecting a target second machine learning model that matches the speech generation information from multiple candidate second machine learning models includes: Based on the speech generation information and the application frequency of the target timbre within a predetermined time period, the target second machine learning model is selected from the plurality of candidate second machine learning models.

4. The method according to claim 3, characterized in that, The step of selecting a target second machine learning model that matches the speech generation information from a plurality of candidate second machine learning models, based at least on the speech generation information, includes: Determine whether the speech generation information includes attribute information, wherein the attribute information indicates the timbre style associated with an object of a preset type, and different attribute information corresponds to different preset types; In response to the speech generation information not including the attribute information, a second machine learning model of the first type is selected from the plurality of candidate second machine learning models as the target second machine learning model; or In response to the speech generation information including the attribute information, and based on the application frequency, a second type of second machine learning model is selected from the plurality of candidate second machine learning models as the target second machine learning model. The first type of second machine learning model corresponds to the first timbre set, the second type of second machine learning model corresponds to the second timbre set, and the second timbre set is a subset of the first timbre set.

5. The method according to claim 4, characterized in that, The step of selecting a second type of second machine learning model from the plurality of candidate second machine learning models based on the application frequency, as the target second machine learning model, includes: Based on the application frequency and whether the second type of second machine learning model is available, a third type or a fourth type of second machine learning model is selected from the plurality of candidate second machine learning models as the target second machine learning model. The third type of second machine learning model corresponds to multiple timbres, and the fourth type of second machine learning model corresponds to a single timbre.

6. The method according to claim 5, characterized in that, The step of selecting a third type or a fourth type of second machine learning model from the plurality of candidate second machine learning models as the target second machine learning model includes: In response to the application frequency reaching a first threshold, based on whether the second machine learning model of the second type is available, a second machine learning model is selected from the third type or the fourth type of second machine learning model as the target second machine learning model; or In response to the application frequency not reaching the first threshold, the third type of second machine learning model is selected from the second type of second machine learning model as the target second machine learning model.

7. The method according to claim 6, characterized in that, Selecting a second machine learning model from the third type of second machine learning model or the fourth type of second machine learning model as the target second machine learning model includes: In response to the inclusion of a second machine learning model for the target timbre in the fourth type of second machine learning model, based on the availability of the second machine learning model for the target timbre, the second machine learning model for the target timbre is selected as the target second machine learning model; or In response to the fact that the second machine learning model of the fourth type includes a second machine learning model for the target timbre, and based on the fact that the second machine learning model for the target timbre is unavailable, the second machine learning model of the third type is selected as the target second machine learning model.

8. The method according to claim 5, characterized in that, The method further includes: In response to the second machine learning model of the fourth type not including a second machine learning model for the target timbre or the second machine learning model for the target timbre being unavailable, a second machine learning model for the target timbre is trained; and In response to the completion of training of the second machine learning model for the target timbre, the second machine learning model for the target timbre is switched from the unavailable state to the available state.

9. The method according to claim 5, characterized in that, The fourth type of second machine learning model includes a third machine learning model, the third machine learning model being designed for the first timbre, and the method further includes: In response to determining that the third machine learning model is in an available state and the application frequency of the first timbre has not reached a third threshold, the third machine learning model is switched from the available state to an unavailable state, wherein the third machine learning model maintains the unavailable state for a preset duration.

10. The method according to claim 1, characterized in that, The step of processing speech generation information using a first machine learning model to obtain intermediate feature representations includes: The speech generation information is processed using the first machine learning model to obtain the intermediate outputs of multiple intermediate layers of the first machine learning model. Select at least one intermediate output of a specified layer from the intermediate outputs of the plurality of intermediate layers; and The intermediate feature representation is determined based on the intermediate output of the at least one specified layer.

11. The method according to claim 10, characterized in that, The first machine learning model includes an autoregressive sub-model and a diffusion sub-model, and the plurality of intermediate layers include at least one layer of the autoregressive sub-model and at least one layer of the diffusion sub-model.

12. The method according to claim 1, characterized in that, The target second machine learning model is configured to generate at least a portion of the multiple speech feature representations in parallel.

13. The method according to claim 1, characterized in that, Each of the multiple speech feature representations is represented as a floating-point vector.

14. The method according to claim 1, characterized in that, Also includes: The speech feature representations are verified sequentially according to the order of the speech feature representations, where the order of the speech feature representations corresponds to the order of the audio segments corresponding to each speech feature representation in the audio. The at least one speech feature used to generate the at least one audio segment is a speech feature representation that has been verified among the plurality of speech feature representations.

15. The method according to claim 14, characterized in that, Also includes: In response to a second speech feature representation among the plurality of speech feature representations failing validation, using the first machine learning model, one or more speech feature representations are generated based on a subset of speech feature representations that have passed validation prior to the second speech feature representation among the plurality of speech feature representations. These one or more speech feature representations are used to replace the second speech feature representation and all subsequent speech feature representations among the plurality of speech feature representations. One or more audio segments are generated based on the one or more speech feature representations, wherein the audio includes the at least one audio feature segment and the one or more audio feature segments.

16. A speech generation device, characterized in that, Includes a module for implementing the speech generation method as described in any one of claims 1 to 15.

17. An electronic device, characterized in that, include: At least one processor; as well as At least one memory, the at least one memory being coupled to the at least one processor and storing instructions for execution by the at least one processor, characterized in that, when executed by the at least one processor, the instructions cause the electronic device to perform the method according to any one of claims 1 to 15.

18. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that, The computer-executable instructions can be executed by a processor to implement the method according to any one of claims 1 to 15.

19. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 15.

Images