Speech synthesis method and electronic equipment

Abstract

The invention provides a speech synthesis method and electronic equipment, and the method comprises the steps: obtaining a first feature inputted by a speech synthesis model, the first feature being obtained based on multi-modal data processing; obtaining a second feature generated by the speech synthesis model based on the first feature reasoning, wherein the second feature is a speech semantic feature associated with the speech acoustic attribute; based on the first feature and the second feature, performing iterative prediction through a reasoning prediction model to obtain at least one candidate acoustic unit of respective corresponding output positions of continuous target iteration prediction; based on the plurality of candidate acoustic units predicted by continuous target iterations, screening a plurality of target acoustic units with continuously adjacent output positions through a speech synthesis model to synthesize target speech; wherein the model parameters of the reasoning prediction model are smaller than the model parameters of the speech synthesis model.

Description

Technical Field

[0001] This application primarily relates to the field of artificial intelligence technology, and more specifically to a speech synthesis method and electronic device. Background Technology

[0002] Currently, in various text-to-speech (TTS) applications, the most common approach is to use autoregressive speech synthesis models based on language models (also known as TTS models). With their powerful long-context modeling capabilities, these models can not only generate prosodic and highly natural speech, but also replicate the speaker's timbre with a small amount of audio input. This better meets the needs of multiple speech tasks such as emotion recognition and speaker recognition, laying the foundation for building a unified interactive model with natural communication capabilities.

[0003] However, in practical applications, due to the significant increase in the sequence size of speech inference processing, and the fact that autoregressive speech synthesis models typically adopt the Transformer architecture, where the computational complexity of the attention mechanism increases quadratically with the sequence length, the continuous expansion of the model parameter size further exacerbates the inefficiency. As a result, the autoregressive inference process is computationally intensive and time-consuming, making it difficult to meet the low latency requirements of scenarios such as real-time interaction. Summary of the Invention

[0004] To address the aforementioned technical problems, this application provides the following technical solutions:

[0005] The first aspect of this application provides a speech synthesis method, including:

[0006] The first feature is obtained from the input of the speech synthesis model, which is based on multimodal data processing;

[0007] The second feature, which is a speech semantic feature associated with the acoustic attributes of speech, is obtained by the speech synthesis model based on the first feature inference.

[0008] Based on the first and second features, iterative prediction is performed through an inference prediction model to obtain at least one candidate acoustic unit corresponding to the output position of each successive target iteration prediction.

[0009] Based on multiple candidate acoustic units predicted in successive target iterations, the speech synthesis model filters out multiple target acoustic units whose output positions are continuously adjacent in order to synthesize target speech.

[0010] The model parameters of the inference prediction model are smaller than those of the speech synthesis model.

[0011] Optionally, the inference parameters used to generate acoustic units are the same between the inference prediction model and the speech synthesis model, so as to unify the prediction probability distribution of acoustic units by the inference prediction model and the speech synthesis model.

[0012] Iterative prediction using an inference prediction model yields at least one candidate acoustic unit at the corresponding output position for each of the successive target iterations, including:

[0013] By performing the nth iteration prediction through the inference prediction model, at least one acoustic unit to be processed is obtained.

[0014] Based on the candidate acoustic units obtained from the first n-1 iterations of prediction, the acoustic units to be processed are deduplicated to obtain at least one candidate acoustic unit corresponding to the output position of each of the consecutive n iterations of prediction.

[0015] The output position of at least one candidate acoustic unit predicted in the nth iteration is adjacent to the output position of at least one candidate acoustic unit predicted in the (n-1)th iteration.

[0016] Optionally, iterative prediction via an inference prediction model also includes:

[0017] Based on the pre-configured repetition penalty parameter, the prediction probability of the same acoustic unit as each candidate acoustic unit predicted in the previous n-1 iterations is reduced in the nth iteration, so that the nth iteration is based on the adjusted prediction probability.

[0018] Among them, the repetition penalty parameter can be determined based on the quality requirements of the target speech in the speech synthesis scenario.

[0019] Optionally, iterative prediction via an inference prediction model also includes:

[0020] Initialize a two-dimensional array to store candidate acoustic units;

[0021] The candidate acoustic units remaining after each iteration of prediction deduplication are stored in the same row of a two-dimensional array;

[0022] When the number of iterations reaches the target number, each candidate acoustic unit stored in the two-dimensional array is input into the speech synthesis model for probability verification;

[0023] Based on the probability verification results, multiple target acoustic units with consecutively adjacent output positions are selected.

[0024] Optionally, the second feature is a multi-level feature set with different speech representation granularities generated by the speech synthesis model decoding the first feature layer by layer.

[0025] Based on the first and second features, iterative prediction is performed using an inference prediction model to obtain at least one candidate acoustic unit at the corresponding output position for each of the consecutive target iterations, including:

[0026] Based on the first and second features, the decoded output features are obtained through an inference prediction model;

[0027] The decoded output features are continuously predicted by the inference prediction model to obtain at least one candidate acoustic unit in each iteration.

[0028] Optionally, based on the first and second features, the decoded output features are obtained through an inference prediction model, including:

[0029] The first feature is transformed by the inference prediction model to obtain the first intermediate feature that matches the feature dimension of the inference prediction model.

[0030] The second feature is fused using the inference prediction model to obtain a second intermediate feature that matches the feature dimensions of the inference prediction model.

[0031] The first and second intermediate features are fused and decoded using an inference prediction model to obtain the decoded output features.

[0032] Optionally, the inference prediction model has the same prediction output module and acoustic unit vocabulary as the speech synthesis model. The prediction output module can output all candidate acoustic units predicted for consecutive target iterations based on the inference parameters and the received decoded output features.

[0033] The model parameters of the fully connected module for obtaining the first intermediate feature, the mapping processing module for obtaining the second intermediate feature, and the fusion decoding module in the inference prediction model are all obtained based on the speech synthesis model.

[0034] Optionally, the inference parameters are determined based on the inference performance requirements in the speech synthesis scenario, which include at least one of inference speed and inference diversity; the inference parameters determined based on inference speed are less than the inference parameters determined based on inference diversity.

[0035] Optionally, multimodal data includes at least two combinations of video data, audio data, and text data;

[0036] The first feature includes visual hidden features obtained by encoding video data, audio hidden features obtained by encoding audio data, and text hidden features and text encoding obtained by encoding text data;

[0037] The second feature includes: high-granularity features, medium-granularity features, and low-granularity features;

[0038] High-granularity features represent the acoustic style information of speech;

[0039] Medium-granularity features characterize the prosodic temporal information of speech;

[0040] Low-granularity features represent the basic acoustic articulation information of speech.

[0041] A second aspect of this application provides an electronic device, which includes at least one memory and at least one processor, wherein:

[0042] A processor loads multiple computer instructions stored in memory, executes the computer instructions, and performs the following steps:

[0043] The first feature is obtained from the input of the speech synthesis model, which is based on multimodal data processing;

[0044] The second feature, which is a speech semantic feature associated with the acoustic attributes of speech, is obtained by the speech synthesis model based on the first feature inference.

[0045] Based on the first and second features, iterative prediction is performed through an inference prediction model to obtain at least one candidate acoustic unit corresponding to the output position of each successive target iteration prediction.

[0046] Based on multiple candidate acoustic units predicted in successive target iterations, a speech synthesis model is used to select multiple target acoustic units whose output positions are continuously adjacent in order to synthesize target speech. Attached Figure Description

[0047] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0048] Figure 1 A schematic diagram of the system architecture for implementing traditional speech synthesis methods;

[0049] Figure 2 This is a schematic flowchart of the speech synthesis method proposed in Embodiment 1 of this application;

[0050] Figure 3 A schematic diagram of the structure of the speech synthesis model for implementing the speech synthesis method proposed in this application embodiment;

[0051] Figure 4 This is a schematic flowchart of the speech synthesis method proposed in Embodiment 2 of this application;

[0052] Figure 5 A schematic diagram of the model architecture for implementing the speech synthesis method proposed in this application embodiment;

[0053] Figure 6 This is a flowchart illustrating the speech synthesis method proposed in Embodiment 3 of this application;

[0054] Figure 7 The intention of storing deduplicated candidate acoustic units in the speech synthesis method proposed in the embodiments of this application through a two-dimensional array during the iterative prediction process;

[0055] Figure 8 This is a schematic flowchart of the speech synthesis method proposed in Embodiment 4 of this application;

[0056] Figure 9 This is a schematic diagram of the speech synthesis device proposed in the embodiments of this application;

[0057] Figure 10 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0058] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is only for explaining specific embodiments and is not intended to limit the application. The embodiments of this application are described below with reference to the accompanying drawings. It will be understood by those skilled in the art that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0059] The terms "first," "second," etc., used in the context of this application and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0060] It is understood that before using the technical solutions disclosed in the embodiments of this application, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this application in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained. For example, in response to receiving a user's active request, a pop-up window may be used, and a textual prompt message may be presented in the pop-up window to clearly inform the user that the requested operation will require the acquisition and use of the user's personal information. The user can choose whether to provide personal information to the electronic device, application, server, or storage medium or other software or hardware that performs the operation of the technical solution of this application based on the prompt message. This application does not limit the prompt message and the method of user authorization implementation. The data involved in this technical solution (including but not limited to the data itself, the acquisition or use of data) shall comply with the requirements of relevant laws and regulations and related provisions.

[0061] Additionally, it should be noted that the model involved in this application can be a general AI (Artificial Intelligence) model with a huge number of parameters (usually reaching billions or even trillions). It can adopt, but is not limited to, Transformer or its architectural variants (such as using only encoder-only / decoder-only, encoder-decoder, or MoE (Mixture of Experts, a neural network architecture) or other basic architectures, and learn the features and rules of natural language by training on a large amount of diverse data, thereby being able to understand and generate natural language.

[0062] In practical applications of this application, AI models may include, but are not limited to, generative models and generative language models (GLMs). Examples include one or more of the following: large language models (LLMs), GPT (Generative Pre-trained Transformer) series models, T5 (Text to Text Transfer Transformer) models, large visual models, and multimodal large models. Depending on actual needs, the models involved in the embodiments of this application may also be expert large models fine-tuned from general AI models based on application requirements, or lightweight models obtained by compressing AI models (e.g., quantization, knowledge distillation, or pruning) to meet the needs of edge deployments with limited computing resources. This application does not limit the model types of the models mentioned in the context.

[0063] Regarding the content described in the background technology section, for example... Figure 1 The diagram shows the structure of a traditional autoregressive speech synthesis model, which typically includes four core components: a text encoder, a speech encoder, an autoregressive language model, and a speech decoder. The text encoder converts the input text into a high-dimensional semantic representation; the speech encoder discretizes continuous audio into a sequence of speech tags; the autoregressive language model processes the tags to predict subsequent acoustic units; and the speech decoder ultimately reconstructs a high-fidelity speech waveform to achieve a speech synthesis effect close to natural human speech. However, the length of the speech tag sequence output by the speech encoder is much greater than that of the text tag sequence, leading to a significant increase in the sequence regularity of the entire speech inference processing, which in turn leads to a sharp increase in the computational cost of the attention mechanism, greatly reducing inference efficiency. To address this, this application proposes a new speech synthesis method, apparatus, and electronic device. The speech synthesis method proposed in this application will be described in detail below with reference to the accompanying drawings.

[0064] Reference Figure 2 This is a flowchart illustrating the speech synthesis method proposed in Embodiment 1 of this application. This method can be applied to electronic devices, which can be device nodes in systems such as servers or cloud platforms, or terminal devices such as smartphones, laptops, or smart home devices; there are no limitations on this. The method proposed in this application can be implemented through applications with data processing capabilities running on the terminal device or cloud services provided by cloud servers, such as… Figure 2 As shown, the speech synthesis method proposed in this embodiment may include, but is not limited to:

[0065] Step S21: Obtain the first feature input from the speech synthesis model, which is obtained based on multimodal data processing;

[0066] Step S22: Obtain the second feature generated by the speech synthesis model based on the first feature inference. The second feature is a speech semantic feature that is associated with the acoustic attributes of speech.

[0067] Step S23: Based on the first and second features, iterative prediction is performed using an inference prediction model to obtain at least one candidate acoustic unit at the corresponding output position for each of the consecutive target iterations.

[0068] Step S24: Based on multiple candidate acoustic units predicted in consecutive target iterations, multiple target acoustic units with continuously adjacent output positions are selected through a speech synthesis model to synthesize target speech.

[0069] In order to solve such Figure 1The speech synthesis model shown above suffers from low inference efficiency due to its inability to output only one acoustic unit at a time. This application proposes a guessing-decoding approach: a fast draft model first "guesses" the subsequent multiple frames, and then a powerful target model performs "parallel review and correction" on these guesses at once. The target model is the original high-quality main model, responsible for the accuracy of the final output. The draft model is a lightweight, low-latency autoregressive model that quickly generates candidate sequences for the target model to verify in parallel, thereby significantly improving the inference generation speed of the AR (Autoregressive) speech synthesis model.

[0070] Therefore, this application uses existing speech synthesis models in the current speech synthesis scenario as the target model, and designs an inference prediction model as a draft model. The model parameters of this inference prediction model are smaller than those of the speech synthesis model. In other words, the speech synthesis model is usually a large multimodal model, while the inference prediction model is a lightweight model. It outputs candidate acoustic units (codec tokens) for multiple consecutive output positions (one output position can correspond to one time point) in a single inference, and transmits them to the speech synthesis model for verification and screening. This avoids the large-scale speech synthesis model generating acoustic units for all output positions from scratch. This application greatly reduces the amount of computation and reduces latency.

[0071] Since draft models for language models are not suitable for speech synthesis scenarios, directly using an LLM draft model as an inference prediction model can easily introduce text semantic bias, leading to problems such as harsh speech and prosodic incoherence. To address this, this application specifically designs an inference prediction model for speech synthesis scenarios. This involves pre-training an inference prediction model adapted to the speech synthesis scenario, enabling it to process hidden features obtained from modal data encoding other than text tokens. This application does not restrict the structure of its inference prediction model or its training implementation method; the model structure described below can be referenced, but is not limited to.

[0072] Based on the above analysis, the speech synthesis model can be a large pre-trained model with many parameters and strong capabilities, such as a multimodal / cross-modal large model. Existing speech synthesis models can be flexibly selected as the target model according to the needs of the actual application scenario. To quickly obtain at least one candidate acoustic unit corresponding to each of multiple consecutive output positions using a lightweight inference prediction model, this application proposes using the corresponding features from the input and inference-generated features of the speech synthesis model as the input of the inference prediction model, and performing consecutive target iterations of prediction to obtain at least one candidate acoustic unit for each output position in each iteration.

[0073] Therefore, the first feature obtained in step S21 and the second feature obtained in step S22 are both inputs to the inference prediction model. The first feature can be a feature of the input end of the speech synthesis model. It can be obtained by processing the original multimodal data in the speech synthesis scenario, such as hidden features obtained by encoding different modal data. This provides a rich initial feature representation for speech synthesis that contains diverse control information, and provides a basic input for generating high-quality, personalized speech.

[0074] This application leverages the powerful representation and learning capabilities of large-scale speech synthesis models to extract core, compact intermediate representations from multimodal inputs to drive speech synthesis. These representations serve as the second feature. The first feature can be input into the speech synthesis model, which then performs inference. The semantic features of the related acoustic attributes generated during this inference process are then determined as the second feature. For example, through the deep network of the speech synthesis model, semantic and acoustic cues directly related to speech generation can be extracted from the noisy and complexly correlated first features, forming a second feature with higher purity. Therefore, the second feature input into the inference prediction model of this application contains content information (what was said) and high-level phonetic information (such as phoneme and syllable-level structure), and is associated with acoustic attributes. It is closely related to the acoustic performance of the final speech (such as spectrum, fundamental frequency, and duration), and can serve as a bridge connecting abstract semantics and concrete sound.

[0075] Subsequently, the first and second features can be simultaneously input into the inference prediction model, providing it with rich and accurate context. This improves the naturalness and alignment accuracy of speech in multimodal scenarios (such as precise matching of virtual human speech and lip movements). The inference prediction model fuses the multiple feature vectors contained in each of the two input features to generate decoded output features. Then, it iteratively predicts the target number of times (e.g., m times, where m can be determined based on the performance of the inference prediction model and / or the resource availability of its deployment device). In each iteration, it predicts at least one candidate acoustic unit (such as an acoustic feature vector, symbol unit, etc., which can be the atomic unit constituting speech, represented as a Codec Token) that conforms to the acoustic properties of speech at a future output position. This iteration is repeated to predict the target number of predictions, thus predicting at least one candidate acoustic unit corresponding to each of the m consecutive future output positions, such as predicting m consecutive future frames. This application does not detail the iterative prediction process within the inference prediction model. As can be seen, this application generates multiple possible speech acoustic evolution paths (candidate acoustic units) quickly with extremely low computational cost, iteratively predicts and simulates the temporal continuity of speech, and significantly reduces the computational burden of subsequent speech synthesis models.

[0076] Because inference prediction models are less powerful than speech synthesis models, while they can achieve efficient inference prediction, they may lack accuracy. Specifically, the candidate acoustic units (candidate pool) predicted for each of the m consecutive output positions (e.g., m time points) may or may not contain the correct acoustic units. Speech synthesis models, on the other hand, possess powerful understanding and generation capabilities. In this case, the speech synthesis model can act as a discriminator for final quality control and refinement. It can accurately select the most suitable target acoustic units from the candidate acoustic units corresponding to each output position provided by the inference prediction model, thereby quickly obtaining a series of continuous and most suitable target acoustic units. This ensures the coherence, naturalness, and accuracy of multiple target acoustic units, improving the quality of the synthesized target speech. This application does not restrict the selection method for target acoustic units, but prefers parallel selection to improve speech synthesis efficiency.

[0077] In summary, compared to traditional speech synthesis methods that directly use large-scale speech synthesis models for autoregressive or parallel generation, although the synthesized target speech has high quality, the inference speed is slow and the computational resources are large. To address this, this application introduces a paradigm of large and small model collaboration (the large model refers to the speech synthesis model, and the small model refers to the inference prediction model). The time-consuming generation task, which requires covering a huge search space (typically accounting for about 70% of the total time in the entire speech synthesis process), is decomposed into two parts: the inference prediction model is responsible for "fast search": leveraging its advantages of fewer model parameters and faster inference, it explores multiple possible paths at low cost (generating at least one candidate acoustic unit for each output position, and selecting multiple candidate acoustic units). The speech synthesis model is responsible for "precise localization": it only needs to filter and decide within a smaller set of candidate acoustic units (also called a candidate pool, rather than the entire space), leveraging its strong understanding and judgment capabilities to retain target acoustic units that are continuously adjacent at the output position and meet the speech quality requirements, in order to reconstruct a continuous speech waveform.

[0078] Therefore, this application cleverly balances computational cost and synthesis quality by designing a three-stage pipeline: a large model provides feature guidance, a small model generates efficient candidate acoustic units, and the large model performs the final high-quality selection. This avoids full-sequence autoregressive computation of the target model and significantly improves the inference speed of speech synthesis with almost no loss of speech quality, reduces real-time synthesis latency, and is more suitable for deployment on resource-constrained edge devices or high-concurrency service scenarios. Furthermore, although the inference prediction model has limited capabilities during iterative prediction, its strong guidance from the first and second features ensures that the predicted candidate acoustic units have high relevance and usability. This allows the speech synthesis model to ultimately select from a high-quality candidate pool, avoiding waste on low-quality paths, resulting in more accurate and efficient decision-making. The complementary advantages of the large and small models make this approach potentially more robust than direct generation by a single model.

[0079] The first feature, derived from multimodal data processing, serves as a carrier of multimodal information fusion, injecting global prior knowledge and control intent into the synthesized speech. This enables the method in this application to naturally support personalized, emotional, and multi-style speech synthesis. In other words, the first feature provides rich context, ensuring a "clear objective" from the outset of speech synthesis. It not only defines "what to say" (speech content) but also "who is saying it" and "how and with what emotion" (speech style and attributes), laying a solid foundation for generating personalized and expressive speech. Furthermore, as source information, the first feature permeates all subsequent steps, ensuring that the final synthesized speech maintains a high degree of consistency with the initial settings in style, timbre, and emotion, achieving precise and controllable multimodal synthesis.

[0080] The second feature is a "speech semantic feature" extracted from the first feature through in-depth processing by a powerful speech synthesis model. It is no longer broad multimodal information, but rather transformed into a high-level abstract representation that is strongly correlated with the acoustic properties of speech and closer to the essence of speech generation. It acts as a "precise bridge" connecting the abstract intent (first feature) and the specific acoustic implementation (candidate acoustic units). Thus, when this high-quality guiding feature (second feature) is input together with the first feature into an inference prediction model with fewer parameters, it greatly compensates for the insufficient representational capabilities of the lightweight inference prediction model, ensuring that it makes fast and effective predictions in the correct direction and guaranteeing the overall quality of the candidate acoustic units.

[0081] It is evident that the first and second features, serving as inputs to the inference prediction model, form a complementary and synergistic driving force. The first feature, acting as a multimodal prior source, ensures the global consistency of the synthesized target speech (overall style, speaker consistency, and controllability). The second feature, acting as a guide for refining the large model, accurately associates high-level semantics with acoustic attributes, providing crucial guidance for the efficient prediction of the subsequent lightweight inference prediction model. It also guides the micro-accuracy of the speech synthesis process (phoneme transitions, prosodic details). The combination of these two features achieves the fusion of global and local information, improving the scene adaptability and naturalness of the synthesized speech. Furthermore, this dual-input mechanism, without sacrificing speech quality, enables the lightweight inference prediction model to produce high-quality candidate units while maintaining low computational cost, reducing the burden on the subsequent speech synthesis model during the selection stage and improving the efficiency and reliability of the entire inference process.

[0082] In some embodiments, the speech synthesis model can be a large model of a multimodal speech generation architecture, such as... Figure 3 The diagram shows the structure of a multimodal large model. The original multimodal data is converted into hidden features that the model can understand through a multimodal encoder (such as a separate visual encoder, audio encoder, text encoder, etc.). These hidden features include visual hidden features (which can capture visual information such as actions and lip movements in the picture, and are used for accurate alignment between speech and vision), audio hidden features (which can extract acoustic information such as timbre and prosody, and support speaker timbre transfer and emotional style adaptation), and text hidden features. It can be understood that the text encoder processes the input text data and outputs text hidden features and text tokens (which provide the semantic content basis for speech synthesis) in addition to outputting text hidden features.

[0083] After that, as Figure 3As shown, the aforementioned hidden features are input into the multimodal understanding module. Through bidirectional context modeling (forward / backward propagation arrows), unified multimodal hidden features are generated, thereby achieving deep fusion of cross-modal information and providing accurate multimodal constraints for subsequent speech generation. The multimodal hidden features are input into the autoregressive speech generation module, which generates Codec Tokens (acoustic units of speech, i.e., discretized speech segment markers) token by token through an autoregressive approach. During this generation process, the Codec Hidden (which is the hidden feature corresponding to the Codec Token) can be maintained, and the sequence length alignment is ensured by padding features (Pad Token / Pad Hidden), ultimately outputting a continuous Codec Token sequence. After streaming decoding into a continuous speech waveform, the target speech is generated segment by segment without waiting for the generation of the complete token sequence, significantly reducing latency and adapting to real-time interactive scenarios. Moreover, based on the understanding capabilities of the multimodal large model, it can handle complex contexts (such as long texts and multimodal semantic associations), and the generated speech has better prosodic coherence and style consistency. It should be noted that the speech synthesis model of this application includes, but is not limited to, Figure 3 The multimodal large model with the structure shown is only used as an example in this application for illustration.

[0084] Based on the above analysis, in some embodiments, the second feature is a multi-level feature set with different speech representation granularities generated by the speech synthesis model decoding the first feature layer by layer. In this way, during the iterative prediction process of the inference prediction model, the decoding output feature can be generated by the inference prediction model based on the first and second features, and the decoding output feature can be continuously predicted by the inference prediction model to obtain at least one candidate acoustic unit for each iteration.

[0085] Based on the descriptions of the first and second features above, the first feature is obtained from multimodal data processing and contains rich and diverse control information. The second feature is a multi-layered feature set with various speech representation granularities generated by the speech synthesis model. It provides precise speech-specific constraints for subsequent inference and prediction models, avoiding the speech stiffness and prosodic inconsistencies caused by relying solely on text features. This improves the quality and validation pass rate of candidate acoustic units and reduces redundant computations in the speech synthesis model.

[0086] Furthermore, after inputting the first and second features into the inference prediction model, the two types of features are aligned and fused using internal fully connected layers and mapping layers to generate decoded output features. This adapts to different speech representation granularities across multiple feature layers, preserving complete speech acoustic association information and providing high-quality input for subsequent iterative predictions, thus improving the accuracy of candidate acoustic units. Moreover, this application uses a lightweight inference prediction model to quickly obtain at least one candidate acoustic unit corresponding to each of multiple consecutive output positions, replacing the inefficient token-by-token generation mode of speech synthesis models, significantly reducing the computational burden of speech synthesis models.

[0087] Based on the above analysis, referring to Figure 4 This is a flowchart illustrating the speech synthesis method proposed in Embodiment 2 of this application. This embodiment describes a possible implementation method for iteratively predicting candidate acoustic units by inputting the first and second features into the inference prediction model. Figure 4 As shown, the implementation method may include:

[0088] Step S41: The first feature is transformed in dimension by the inference prediction model to obtain the first intermediate feature that matches the feature dimension of the inference prediction model.

[0089] Step S42: The second feature is fused using the inference prediction model to obtain a second intermediate feature that matches the feature dimension of the inference prediction model.

[0090] Based on the descriptions of the sources and contents of the first and second features above, the inference prediction model first performs feature dimension transformation on both features during the alignment and fusion process to obtain the hidden dimensions required by the inference prediction model, and then performs the fusion process. (Refer to...) Figure 5 The diagram shows the structure of an inference prediction model. The various features contained in the first feature are used as input to the mapping layer in the inference prediction model. These features are compressed to the hidden dimension k (adapted feature dimension) required by the inference prediction model, while keeping all the first features without loss. This lossless dimension processing enhances the input representation of the inference prediction model instead of drastically increasing the computational cost.

[0091] As can be seen, the mapping processing module is a lossless dimension transformation module, which can perform linear mapping (such as dimension compression) on the first feature, solve the problem of feature dimension adaptation between the first feature and the inference prediction model, and ensure that all effective information of multimodal modes in the first feature (such as visual lip shape, reference audio timbre, etc.) is not lost or distorted, providing complete input constraints for subsequent fusion.

[0092] Furthermore, the multiple features contained in the second feature are used as inputs to the fully connected module (FC Layer) in the inference prediction model to compress them to the hidden dimension k required by the inference prediction model. At the same time, information from different levels in these features is fused, such as linearly fusing and enhancing multi-level speech semantic features, extracting the core acoustic information of each level, and mapping it to the feature dimension of the inference prediction model. In other words, the layered speech semantic features are integrated into a feature vector of a unified dimension. Similarly, the acoustic properties specific to speech are enhanced, avoiding information redundancy caused by the dispersion of multi-level features, and providing accurate speech constraints for subsequent fusion. That is, the input representation power of the inference prediction model is enhanced, rather than the computational cost is increased dramatically.

[0093] Step S43: The first intermediate feature and the second intermediate feature are fused and decoded using an inference prediction model to obtain the decoded output feature;

[0094] In this embodiment, based on the above description of the first feature, the second feature, and the intermediate features obtained from their respective processing, step S43 can be performed by the cross-modal fusion decoding module (Decoder Layer) in the inference prediction model, processing the first intermediate feature (token embedding, which is a type of hidden feature, denoted as g) output by the Mapping Layer. map (Hidden)), and the second intermediate feature output by the FC Layer (the fused speech and semantic features, which can be denoted as g). fc (Hidden)) performs alignment and deep fusion (e.g., concatenating two intermediate features before inputting them (DecoderLayer, etc.) to achieve precise binding of multimodal information and speech semantics, generating decoded output features that combine multimodal constraints and speech acoustic properties (which can be denoted as δ). msd (Feature) allows candidate acoustic units generated by subsequent iterations to not only meet the requirements of multimodal input scenarios (such as visual lip alignment) but also possess natural speech acoustic properties (such as prosody and timbre).

[0095] Optionally, the fusion decoding module can adopt a Transformer architecture with a multi-layer network structure. Its core layer is the Transformer decoder layer, which generates the next output, i.e., the decoded output feature (hidden state), by performing self-attention calculation on the first and second intermediate features. However, it is not limited to this. This application requires the pre-trained modules (such as...) that constitute the inference prediction model. Figure 5 The structure of the modules labeled with the flame pattern shown (which need to be trained) is not restricted.

[0096] Step S44: The decoded output features are continuously predicted by the inference prediction model to obtain at least one candidate acoustic unit for each iteration.

[0097] For details on the implementation of step S44, please refer to the description of the corresponding part of the above embodiment.

[0098] In summary, the inference prediction model, through a refined process of feature-specific processing and cross-modal fusion decoding, achieves precise adaptation between features and the inference prediction model while ensuring the complete preservation of multimodal information and speech semantics. Specifically, the lossless dimensionality transformation of the first feature ensures that the visual and audio information of the multimodal input is not lost, providing complete scene constraints for subsequent speech generation; the fusion processing of the second feature integrates speech semantic information at different levels, strengthens the speech-specific acoustic attributes, and improves the naturalness and accuracy of candidate acoustic units. Through cross-modal fusion decoding, the deep binding of multimodal information and speech semantics allows the final generated candidate acoustic units to not only meet the alignment requirements of multimodal scenarios (such as virtual lip-syncing) but also possess natural rhythm and timbre, significantly improving the quality of synthesized speech.

[0099] Moreover, as a lightweight draft model, the inference prediction model maintains low computational complexity while ensuring the accuracy of feature adaptation and fusion through the above-mentioned dedicated processing flow. Combined with the architecture of subsequent candidate acoustic unit generation and target model (speech synthesis model) verification, it fundamentally alleviates the efficiency bottleneck of autoregressive speech synthesis models and achieves a balance between efficiency and speech quality.

[0100] In some embodiments, the final output layer of the inference prediction model, namely the prediction output module, transforms the decoded output features into the predicted probability distribution logits of the acoustic unit (token) at the next output position. Since the acoustic unit vocabulary of the model is generally small, no additional optimization processing is required here. It can directly select at least one candidate acoustic unit for this prediction from the acoustic unit vocabulary based on the predicted probability distribution of the acoustic unit vocabulary according to the preset sampling strategy, such as one or more acoustic units with the highest probability, etc., without any restrictions.

[0101] Optionally, the prediction output module of the aforementioned inference prediction model can be the same as the final output layer in the speech synthesis model (same data structure), such as an LM Head structure. It can be transformed from the input hidden state into a predicted probability distribution for the next token using a linear transformation and normalization function (such as softmax). In this way, during the training of the inference prediction model, the parameters of the inference prediction module are directly frozen, and it does not need to participate in the training. Only the parameters of the fully connected module, the mapping processing module, and the fusion decoding module need to be trained to achieve the aforementioned processing capabilities. That is, the model parameters of the fully connected module used to obtain the first intermediate feature, the mapping processing module used to obtain the second intermediate feature, and the fusion decoding module in the inference prediction model are all trained based on the speech synthesis model.

[0102] It should be understood that, as Figure 5 As shown, during the training process of the inference prediction model, the model parameters of the entire speech synthesis model are frozen, that is, the model parameters are not changed (e.g., ...). Figure 5 (The modules / layers that need to freeze parameters are marked with a snowflake pattern in the middle). According to the speech synthesis method steps described in this application, the sample multimodal data is processed. When the speech synthesis model verifies the candidate sample acoustic unit output by the initial inference prediction model, the loss between the candidate sample acoustic unit and the standard sample acoustic unit generated by the speech synthesis model itself is directly obtained. By minimizing the loss, the model parameters of the initial inference prediction model (the parameters of the fully connected module, the mapping processing module and the fusion decoding module) are adjusted until the loss converges or the preset number of training times is reached, etc., to obtain the trained inference prediction model. However, this method of model training is not limited to this method, and this application does not limit it.

[0103] Furthermore, the inference prediction model and the speech synthesis model share the same acoustic unit vocabulary, such as a preset number of Codec Tokens. Tokens with the same index correspond to the same speech acoustic features, ensuring that the candidate and target acoustic units generated by the two models are completely aligned semantically and in terms of format. This eliminates the need for format or semantic conversion of candidate acoustic units, allowing them to be directly verified by the speech synthesis model and avoiding the computational overhead and information loss associated with cross-model feature mapping. Moreover, the unified vocabulary ensures that the definitions of acoustic units in the inference prediction model and the speech synthesis model are completely consistent, resulting in candidate acoustic units whose quality more closely matches the output of the speech synthesis model. This improves the verification pass rate and avoids quality issues such as prosodic breaks and inaccurate pronunciation.

[0104] The prediction output module, based on inference parameters and received decoded output features, outputs all candidate acoustic units predicted for consecutive target iterations in a single step. This maintains the continuity of speech temporal sequence and prosody, avoids prosodic breaks caused by token-by-token generation, and enhances speech coherence. All candidate acoustic units are then input into the speech synthesis model for batch validation, reducing the number of validations required for the target model, avoiding the quadratic computational complexity of long-sequence autoregression, and significantly improving inference efficiency.

[0105] It should be noted that in the above processing, inference parameters, such as hyperparameters used to control the randomness or determinism of the predicted probability distribution of the model output, i.e., the temperature coefficient, can be applied to the logistic values ​​logits of the model output, and the final predicted probability distribution is obtained through the Softmax function.

[0106] In practical applications, setting the temperature too low (e.g., 0.1-0.5, referred to as low temperature) amplifies the differences between high-probability options, making the predicted probability distribution sharper and more predictable. The model becomes very confident, almost always choosing the word / phoneme / acoustic unit with the highest probability, resulting in a more stable, predictable, and consistent output, but potentially lacking variation and naturalness, appearing monotonous. Setting the temperature too high (e.g., 0.8-1.2, referred to as high temperature) "smooths" the probability distribution, reducing the gap between high-probability and low-probability options, making the distribution more "flat" and "random." This results in a more diverse and creative output, but may introduce incoherence, errors, or even nonsensical statements.

[0107] As can be seen, if temperature=0, always selecting the acoustic unit with the highest probability (greedy decoding), the model will continuously choose the "safest" prolonged or smoothed phonemes, resulting in elongated sounds and stiff rhythm. In other words, greedy decoding compresses the diversity of speech, leading to a monotonous and rigid output. Therefore, when releasing a speech synthesis model, the default setting is temperature=0.9 or another relatively high value.

[0108] However, in the guessing decoding technique, the current draft model has a temperature of 1, resulting in high randomness in the guessed tokens. Using the model's original, unmodified prediction probability distribution, it's easy to make errors after rigorous verification by the target model. Therefore, this application requires a high degree of consistency between the token generation strategies of the target model and the draft model to improve decoding efficiency and accelerate inference, such as using a temperature of 0 or other lower values. If the temperature settings of the two models are different, using the guessing decoding approach to accelerate the inference of the speech synthesis model will not significantly improve decoding efficiency (Acceptance Length), making it difficult to meet the requirements of inference acceleration.

[0109] Therefore, in this application, the inference parameters used to generate acoustic units are the same between the inference prediction model and the speech synthesis model, so as to unify the prediction probability distribution of acoustic units by the inference prediction model and the speech synthesis model, avoid the deviation of the candidate unit probability distribution caused by parameter differences, make the generated candidate units more in line with the probability preference of the speech synthesis model, thereby improving the verification pass rate of candidate acoustic units, that is, improving the quality of candidate acoustic units, reducing the repeated calculation of the speech synthesis model, reducing the overall inference computation and latency, maximizing the efficiency gain of speculative decoding, eliminating the computational cost and information loss of cross-model calibration, simplifying the technical link of dual-model collaboration, reducing the complexity of the overall architecture, and better adapting to the low latency requirements of real-time interactive scenarios.

[0110] Moreover, the uniform probability distribution ensures that the pronunciation, rhythm, timbre and other attributes of the candidate units are highly consistent with the output of the target model, avoiding the problems of harsh speech or rhythmic incoherence caused by probability deviation. The final synthesized target speech is closer to the original output of the target model in terms of naturalness and scene adaptability, maintaining high naturalness while improving efficiency.

[0111] Based on the above analysis, preferably, the inference parameters in the above model can be determined based on the inference performance requirements of the speech synthesis scenario. Inference performance requirements may include at least one of inference speed and inference diversity. Inference parameters determined based on inference speed are less than those determined based on inference diversity. For example, in a speed-priority scenario, the temperature is set to 0.3~0.6, while in a diversity-priority scenario, it is set to 0.8~1.2. This design breaks the limitation of fixed values ​​for inference parameters, achieving precise matching between inference parameters and scenario requirements. Simultaneously, by relying on the quantitative differences in inference parameter values, it takes into account the core performance requirements of different scenarios. For example, in scenarios requiring low latency and high real-time performance, the same temperature setting in both models is lower, or even 0; while in scenarios requiring diversity and creativity, the same temperature setting in both models is higher, in order to select a target acoustic unit that better meets the required style, timbre, and other requirements, thus achieving personalized speech synthesis.

[0112] Therefore, this application achieves precise, scenario-based control of inference performance by binding inference parameters to the inference performance requirements of the speech synthesis scenario and clarifying the quantitative relationship between inference speed and inference diversity. This results in performance synergy with the dual-model collaborative architecture. Specifically, the inference parameters can be dynamically determined based on scenario-based inference performance requirements such as inference speed and inference diversity. This allows the same dual-model speech synthesis architecture to adapt to different scenarios, such as real-time interaction and high-quality non-real-time synthesis, without requiring separate modifications. This overcomes the technical bottleneck of fixed parameters failing to meet the needs of multiple scenarios, significantly improving the practicality and versatility of the technical solution.

[0113] Furthermore, by using smaller inference parameters determined by inference speed, the probability distribution of acoustic units in the model can be centralized, reducing the computation and selection time of candidate units. Combined with the lightweight architecture of the inference prediction model and the design of parallel candidate generation, inference efficiency is improved from both parameter tuning and model design dimensions, meeting the low latency requirements of real-time interactive scenarios such as virtual human dialogue and intelligent voice assistants. On the other hand, by using larger inference parameters determined by inference diversity, the probability distribution of acoustic units in the model can be expanded, generating more diverse combinations of acoustic units. This effectively avoids the problems of mechanical repetition and monotonous prosody caused by low parameters. At the same time, by combining the unified probability distribution and shared vocabulary of the two models, diverse candidate units can still meet the requirements of high-quality speech synthesis, satisfying the naturalness and personalization needs of scenarios such as film and television dubbing and audiobook production.

[0114] Furthermore, this application explicitly states that parameter values ​​based on inference speed are less than those based on inference diversity, thus establishing a quantifiable and reproducible parameter configuration criterion. This provides a clear tuning basis for model deployment, reduces deployment costs, and distinguishes the dynamic configuration scheme of inference parameters from traditional fuzzy adaptive strategies, enhancing the feasibility and innovation of the technical solution. Moreover, the dynamic configuration of inference parameters, combined with designs such as dual-model shared output modules, unified probability distribution, and continuous candidate acoustic units in a single output, creates a performance synergy effect. This maximizes inference efficiency in speed-priority scenarios and ensures speech quality in diversity-priority scenarios, breaking the inherent perception that efficiency and quality are mutually exclusive in existing autoregressive speech synthesis models. It fundamentally achieves a non-mutually exclusive trade-off between inference speed and inference diversity.

[0115] However, when the temperature is set low, especially when temperature=0, prolonged sounds occur, meaning that candidate acoustic units are the same in multiple consecutive output positions. This leads to excessive repetition of the same phoneme or sound segment in the final synthesized target speech. To address this, this application further proposes to add deduplication processing during the iterative prediction process of the inference prediction model without affecting the output speech quality. Specifically, by introducing dynamic deduplication processing based on historical candidate acoustic units during the iterative process of the inference prediction model and strictly constraining the continuity of the output positions of candidate acoustic units, the repetition problem generated by autoregressive iteration is solved, and the temporal coherence requirement of speech synthesis is met, thus achieving performance synergy with the dual-model collaborative architecture.

[0116] Based on this, refer to Figure 6 The flowchart of the speech synthesis method proposed in Embodiment 3 of this application is shown. During the iterative prediction process using the inference prediction model, as shown... Figure 6 As shown, multiple candidate acoustic units predicted in successive target iterations can be obtained through the following steps:

[0117] Step S61: Perform the nth iteration prediction using the inference prediction model to obtain at least one acoustic unit to be processed;

[0118] In this embodiment, the inference prediction model proceeds to the nth round of prediction according to autoregressive iterative logic. Based on the fused decoded output features, it generates at least one candidate acoustic unit corresponding to the nth output position in the current round. This candidate has not undergone duplicate verification and is defined as the acoustic unit to be processed. Here, n is a positive integer greater than 1 and is the round identifier of the iterative prediction. The output position of at least one candidate acoustic unit predicted in the nth iteration is adjacent to the output position of at least one candidate acoustic unit predicted in the (n-1)th iteration, corresponding to "continuous acoustic segment positions" in speech synthesis, ensuring the continuity of speech temporal sequence and avoiding speech prosody breaks caused by output position misalignment.

[0119] Step S62: Based on the candidate acoustic units predicted in the first n-1 iterations, perform deduplication processing on the acoustic units to be processed.

[0120] Step S63: Output at least one candidate acoustic unit for each corresponding output position predicted in n consecutive iterations.

[0121] The inference prediction model directly stores the acoustic units to be processed obtained in the first iteration as candidate acoustic units. Based on the information (hidden state) of the acoustic units to be processed in the first iteration (corresponding to the first future output position), it infers and generates the acoustic units to be processed in the second iteration (corresponding to the first future output position), and then performs deduplication. The deduplication principle is: candidate acoustic units predicted in the first iteration will not be stored repeatedly as candidate acoustic units in the second iteration, and no new candidate acoustic units will be added after deduplication. The deduplication strategy implemented is the same as the deduplication strategy used in the decoding process of the speech synthesis model, and can be directly implemented using the corresponding deduplication functions, without any restrictions.

[0122] Subsequently, during the third iteration of prediction, the inference model uses the information (hidden state) of the acoustic units to be processed in the second iteration to generate the acoustic units to be processed in the third iteration (corresponding to the third future output position), and performs deduplication. That is, the candidate acoustic units corresponding to the first two output positions will not be stored as candidate acoustic units corresponding to the third output position, nor will new candidate acoustic units be added. This process continues until the target number (n) of iterations of prediction are completed, and at least one candidate acoustic unit corresponding to the nth future output position is obtained and stored.

[0123] Therefore, this application achieves soft deduplication in the lightweight inference prediction model stage by performing real-time matching and verification between the nth acoustic unit to be processed and the previous n-1 historical candidate acoustic units. This avoids duplicate acoustic units from entering the subsequent screening process, solving the speech stuttering and mechanical repetition problems that easily occur in autoregressive iterations. At the same time, it reduces invalid verification of the speech synthesis model and improves the overall inference efficiency. Furthermore, it explicitly limits the output positions of the candidate acoustic units in the nth and n-1th iterations to be adjacent, and after deduplication, integrates them into a set of candidate acoustic units for n consecutive iterations. This ensures that the output positions of all candidate acoustic units are distributed in a continuous temporal sequence, perfectly matching the physical property of "continuous and unbroken" speech waveforms. From the perspective of positional constraints, it avoids speech prosody disorder and ensures the coherence of synthesized speech.

[0124] In addition, this application moves the deduplication process to the inference prediction model stage, utilizes the low computational complexity of the small model to complete the duplication verification, and only passes the effective candidate acoustic units to the speech synthesis model, which greatly reduces the screening computation of the speech synthesis model, further amplifies the efficiency gain of the dual-model collaborative architecture, and alleviates the inference delay bottleneck of autoregressive speech synthesis.

[0125] In some embodiments, the deduplication strategy employed in the above deduplication process can be a soft deduplication strategy. This involves reducing the selection probability of acoustic units that overlap with historical candidate acoustic units, rather than directly eliminating them. This avoids duplication while preserving the diversity of candidate acoustic units, thus meeting the naturalness requirements of speech synthesis. The resulting candidate acoustic unit set ensures that each output position corresponds to at least one candidate acoustic unit with no significant overlap, and that the output positions of all candidate acoustic units are distributed in a continuous temporal sequence, perfectly matching the requirements of speech synthesis for generating continuous acoustic segments.

[0126] To address this, this application proposes a pre-configured repetition penalty parameter to reduce the prediction probability of the same acoustic unit as each candidate acoustic unit predicted in the previous n-1 iterations in the nth iteration, so that the nth iteration is based on the adjusted prediction probability. Optionally, the repetition penalty parameter is consistent in both models to balance the repetition suppression of candidate acoustic units with the naturalness of the synthesized speech, which can be determined based on the quality requirements of the target speech in the speech synthesis scenario. For example, in the AR real-time interaction scenario, the quality requirement of the target speech is "low latency + basic fluency", in which case the repetition penalty parameter is 1.2~1.5; in the film and television dubbing scenario, the quality requirement of the target speech is "high naturalness + prosodic matching", in which case the repetition penalty parameter is 1.0~1.2, but is not limited to these values.

[0127] Therefore, this application employs a soft deduplication method that reduces the probability of duplicate candidates, rather than hard deletion. This avoids repetition of acoustic units in consecutive positions while retaining multiple candidate acoustic units in the current round, providing sufficient selection space for the speech synthesis model. It resolves the contradiction between insufficient diversity due to deduplication and excessive repetition due to non-deduplication, balancing inference efficiency and speech naturalness. Furthermore, the soft deduplication logic is compatible with the dynamic configuration requirements of inference parameters. In speed-priority scenarios, it can quickly lock in non-repeating high-probability candidates, further improving iteration speed; in diversity-priority scenarios, it can retain more candidates, matching the diverse requirements of high parameters. This allows the same architecture to adapt to different speech synthesis scenarios without modification, improving the practicality and scenario adaptability of the technical solution.

[0128] In one possible implementation, the candidate acoustic units predicted in each iteration can be stored using a two-dimensional array (Aarry array), such as... Figure 7As shown, the rows of the Aarry array represent the candidate acoustic unit tokens for each layer (one iteration of prediction, corresponding to the same output position), and the columns of the array represent the layer number of each guessed token during the guessing and decoding process. Based on this, refer to... Figure 8 The flowchart of the speech synthesis method proposed in Embodiment 4 of this application, shown below, may further include the following in the iterative prediction process using the inference prediction model described above:

[0129] Step S81: Initialize the two-dimensional array used to store candidate acoustic units;

[0130] Step S82: Store the remaining candidate acoustic units after each iteration of prediction deduplication into the same row of a two-dimensional array;

[0131] Step S83: When the number of iterations reaches the target number, input each candidate acoustic unit stored in the two-dimensional array into the speech synthesis model for probability verification;

[0132] Step S84: Based on the probability verification results, select multiple target acoustic units whose output positions are consecutively adjacent to each other, and use them to synthesize target speech.

[0133] Based on the above analysis, the row numbers of the two-dimensional array correspond one-to-one with the iteration number and the output position; that is, the k-th row, the k-th iteration, and the k-th output position correspond one-to-one. This constraint matches the temporal requirement in speech synthesis that "the output position advances continuously with each iteration." The column dimension corresponds to the remaining candidate acoustic units after deduplication in each iteration. The number of these units can be determined as needed, and each row must have the same number of candidate acoustic units.

[0134] As can be seen, this application provides a structured storage medium for candidate acoustic units in multiple iterations, solving the problems of "location confusion and difficulty in batch retrieval" caused by the scattered storage of candidates in multiple rounds. At the same time, it strongly binds candidate units to output positions, ensuring the traceability of speech timing. In addition, through the structured mapping between the output positions of acoustic units and candidate acoustic units, subsequent verification and screening can directly lock the output position based on the row number, avoiding misalignment between candidate acoustic units and output positions. It also provides a regular input format for batch verification, ensuring accurate matching between probability calculation and output position.

[0135] In this process, after the speech synthesis model completes batch probability verification, it extracts the candidate acoustic unit with the highest probability from each output position (each row of the array). These candidate units, belonging to different positions and possessing the highest probability, are then integrated according to their row numbers from smallest to largest (iteration rounds from earliest to latest) to form a sequence of target acoustic units with consecutively adjacent output positions. The positional continuity of this sequence perfectly matches the temporal continuity of the speech waveform, and is finally input into the speech decoder to reconstruct the target speech. This application does not elaborate on the batch verification process.

[0136] Therefore, the two-dimensional array in this application only stores the valid candidate acoustic units after each round of deduplication, without duplicate or redundant data, which greatly reduces the memory usage of candidate storage. Combined with the efficiency of batch verification, this solution can be adapted to the real-time speech synthesis needs of edge / side resource-constrained scenarios such as smart speakers and virtual human hardware, breaking through the limitations of traditional autoregressive models that are difficult to implement in resource-constrained scenarios.

[0137] Furthermore, the storage structure of this embodiment works synergistically with the iterative soft deduplication, the dual-model shared prediction output module / acoustic unit vocabulary, and the unified inference parameters. This reduces the computational load of batch verification through deduplication and enables zero-cost batch verification and accurate probability determination of candidates through shared modules and unified parameters. This allows the efficiency and quality advantages of the dual-model speculative decoding architecture to be fully utilized, achieving a non-mutually exclusive improvement in speech synthesis inference efficiency and synthesis quality as a whole.

[0138] In addition, as a general standardized data structure, two-dimensional arrays can be directly adapted to the batch computing interface of speech synthesis models without additional format conversion, reducing the engineering development costs of different platforms such as edge and cloud. At the same time, the standardized storage and verification logic makes the technical solution highly implementable and reproducible, which meets the core requirements for the implementation of patented technology and also provides good compatibility for subsequent model upgrades.

[0139] According to the speech synthesis methods described in the embodiments above, when the multimodal data used to obtain the first feature can include at least two combinations of video data, audio data, and text data, the first feature can include a visual hidden feature obtained by encoding the video data (e.g., implemented by a visual encoder), an audio hidden feature obtained by encoding the audio data (e.g., implemented by an audio encoder), and a text hidden feature and a text token obtained by encoding the text data (e.g., implemented by a text encoder). This application does not limit the implementation methods of these feature encodings. It should be understood that since the multimodal data contains different data types, the first feature can include the corresponding encoded hidden features of the corresponding data type.

[0140] In the process of reasoning about the first feature in the speech synthesis model, the generated second feature can include: high-granularity features, medium-granularity features, and low-granularity features. That is, the second feature is a multi-layered speech semantic feature divided according to low / medium / high speech representation granularity. High-granularity features represent the acoustic style information of the speech, such as timbre, emotion, and speaker characteristics, to ensure stylistic consistency in the synthesized speech; medium-granularity features represent the prosodic temporal information of the speech, such as intonation, pauses, and speech rate, to ensure the natural fluency of the synthesized speech; low-granularity features represent the basic acoustic pronunciation information of the speech, such as phonemes and syllables, to ensure the pronunciation accuracy of the acoustic unit prediction. This application synchronously transmits the multi-layered speech semantic features to the inference prediction model to assist it in generating candidate acoustic units that conform to the acoustic properties of the speech.

[0141] The high-granularity, medium-granularity, and low-granularity features mentioned above originate from different layers of the speech synthesis model. Combining the hierarchical characteristics of human speech, subtle acoustic variations (low-granularity) can be used to construct phonemes / phonemes (medium-granularity), which are then combined into syllables, words, and complete intonation and prosody (high-granularity). Therefore, the low-granularity features in this application typically originate from the bottom layer of the speech synthesis model, closely resembling the first feature of the original input or undergoing shallow processing, containing rich local and detailed acoustic information (such as subtle spectral envelopes and instantaneous frequency changes). In this way, low-granularity features can provide the inference prediction model with precise "phoneme-level" or "frame-level" anchor points, enabling the inference prediction model to have very detailed references when predicting each acoustic unit. This allows the model to generate acoustically reliable candidate acoustic units, avoiding unclear pronunciation or acoustic distortion, which directly improves the clarity and naturalness of the synthesized speech.

[0142] Medium-granularity features typically come from the middle layer of the speech synthesis model. They are an intermediate stage of information extraction and integration, containing information such as syllables, word boundaries, and local prosodic structures (e.g. stress and rhythm). These features provide guidance for the inference prediction model to ensure contextual coherence, helping it understand the role of the currently synthesized part in a longer speech segment (e.g., a word or phrase). This results in more smooth and coherent temporal sequences of predicted acoustic units, effectively avoiding abrupt jumps or unreasonable pauses in the candidate acoustic unit sequence and improving fluency.

[0143] High-granularity features, originating from higher levels of the speech synthesis model or deeper layers of the decoder, represent the highest level of abstraction. They encompass global semantics, speaker style, emotional tone, and sentence-level prosodic contours (such as rising intonation in interrogative sentences and falling intonation in declarative sentences). These features provide macro-level control over the global intent and style of the inference prediction model, ensuring that the entire sequence of candidate acoustic units generated by the inference prediction model maintains stylistic and semantic consistency with the final goal of the speech synthesis model. This guarantees the emotional accuracy and overall consistency of the synthesized speech, avoiding "out-of-tune" or "style drift." Therefore, it is evident that traditional autoregressive speech synthesis models typically rely on single textual semantic features or coarse-grained speech features, lacking hierarchical speech-specific representations. This leads to problems such as inaccurate pronunciation, harsh prosody, and inconsistent style in synthesized speech. This application, through hierarchical and systematic distillation of the knowledge within the speech synthesis model, clearly deconstructs the thought process by which the speech synthesis model understands and generates speech. It simultaneously provides the corresponding speech semantic features at three levels to the inference prediction model. This not only greatly empowers the lightweight inference prediction model, enabling its predicted candidate acoustic units to approach the level of larger models in terms of acoustic detail, local coherence, and global style (improving quality), but also, through the combined low-, medium-, and high-granularity multi-layered speech semantic features, constitutes a clear, multi-layered, and strong constraint. This significantly narrows the range of reasonable candidate acoustic units at each prediction step, significantly reducing its confusion regarding the next output position. In other words, it significantly reduces uncertainty in the prediction process, enabling rapid selection within a highly constrained and correct solution space, reducing iterations or computational supplementation, thereby greatly accelerating inference speed while improving speech quality.

[0144] Reference Figure 9 This is a schematic diagram of the speech synthesis device proposed in the embodiments of this application, as shown below. Figure 9 As shown, the speech synthesis device may include:

[0145] The first feature acquisition module 91 is used to acquire the first feature input by the speech synthesis model, which is obtained based on multimodal data processing.

[0146] The second feature acquisition module 92 is used to obtain the second feature generated by the speech synthesis model based on the first feature inference. The second feature is a speech semantic feature associated with the acoustic attributes of speech.

[0147] The iterative prediction module 93 is used to perform iterative prediction based on the first feature and the second feature through an inference prediction model to obtain at least one candidate acoustic unit at the corresponding output position of each successive target iteration prediction.

[0148] The filtering module 94 is used to filter multiple candidate acoustic units based on consecutive target iterations of prediction, and to filter multiple target acoustic units with consecutively adjacent output positions through the speech synthesis model in order to synthesize target speech; wherein, the model parameters of the inference prediction model are smaller than the model parameters of the speech synthesis model.

[0149] Optionally, the inference parameters used to generate acoustic units are the same between the inference prediction model and the speech synthesis model, so as to unify the prediction probability distribution of acoustic units by the inference prediction model and the speech synthesis model. Preferably, the inference parameters are determined based on the inference performance requirements in the speech synthesis scenario, and the inference performance requirements include at least one of inference speed and inference diversity; the inference parameters determined based on inference speed are smaller than the inference parameters determined based on inference diversity.

[0150] In some embodiments, the iterative prediction module 93 may include:

[0151] The first prediction unit is used to perform the nth iteration prediction through the inference prediction model to obtain at least one acoustic unit to be processed.

[0152] The deduplication unit is used to deduplicate the acoustic units to be processed based on the candidate acoustic units predicted in the first n-1 iterations, so as to obtain at least one candidate acoustic unit corresponding to the output position of each of the consecutive n iterations.

[0153] In this case, the output position of at least one candidate acoustic unit predicted in the nth iteration is adjacent to the output position of at least one candidate acoustic unit predicted in the (n-1)th iteration.

[0154] In some embodiments, the iterative prediction module 93 may further include:

[0155] The prediction probability adjustment unit is used to reduce the prediction probability of the same acoustic unit as each candidate acoustic unit predicted in the previous n-1 iterations in the nth iteration based on a pre-configured repetition penalty parameter, so as to perform the nth iteration based on the adjusted prediction probability; wherein, the repetition penalty parameter can be determined based on the quality requirements of the target speech in the speech synthesis scenario.

[0156] In some embodiments, the iterative prediction module 93 may further include:

[0157] An initialization unit is used to initialize a two-dimensional array for storing candidate acoustic units;

[0158] The deduplication storage unit is used to store the remaining candidate acoustic units after each iteration of prediction deduplication into the same row of the two-dimensional array;

[0159] The transmission verification unit is used to input each candidate acoustic unit stored in the two-dimensional array into the speech synthesis model for probabilistic verification when the number of iterations reaches the target number.

[0160] The output unit is used to filter multiple target acoustic units with consecutively adjacent output positions based on the probability verification results.

[0161] Optionally, the second feature mentioned above can be a multi-level feature set with different speech representation granularities generated by the speech synthesis model decoding the first feature layer by layer.

[0162] Based on this, the iterative prediction module 93 may include:

[0163] A decoding processing unit is used to obtain decoded output features based on the first feature and the second feature through an inference prediction model;

[0164] The prediction unit is used to perform consecutive target iterations of prediction on the decoded output features through an inference prediction model, and to obtain at least one candidate acoustic unit for each iteration.

[0165] Optionally, the decoding processing unit may include:

[0166] The first intermediate feature acquisition unit is used to perform dimensional transformation on the first feature through the inference prediction model to obtain the first intermediate feature that matches the feature dimension of the inference prediction model.

[0167] The second intermediate feature acquisition unit is used to fuse the second feature through the inference prediction model to obtain a second intermediate feature that matches the feature dimension of the inference prediction model.

[0168] The fusion decoding unit is used to fuse and decode the first intermediate feature and the second intermediate feature through an inference prediction model to obtain the decoded output feature.

[0169] Optionally, the inference prediction model and the speech synthesis model have the same prediction output module and acoustic unit vocabulary. The prediction output module can output all candidate acoustic units predicted for consecutive target iterations based on the inference parameters and the received decoded output features. The model parameters contained in the fully connected module for obtaining the first intermediate feature, the mapping processing module for obtaining the second intermediate feature, and the fusion decoding module in the inference prediction model are all trained based on the speech synthesis model.

[0170] The multimodal data includes at least two combinations of video data, audio data, and text data. The first feature includes visual latent features encoded from video data, audio latent features encoded from audio data, and text latent features and text encodings encoded from text data. The second feature includes high-granularity features, medium-granularity features, and low-granularity features. High-granularity features represent the acoustic style information of speech; medium-granularity features represent the prosodic temporal information of speech; and low-granularity features represent the basic acoustic articulation information of speech.

[0171] This application also provides a computer program product including computer-readable instructions, which, when executed on a computer device (such as a terminal), cause the electronic device to implement any of the speech synthesis methods provided in this application.

[0172] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by a computer device, the computer device is able to implement any of the speech synthesis methods provided in this application.

[0173] This application also provides an intelligent agent, or intelligent program, to receive multimodal data and implement the speech synthesis method proposed in this application by interacting with a model. The implementation process can be referred to the description of the corresponding part of the method embodiment above. In this implementation process, other components of the application program or operating system can also be controlled through interface calls or other interactive methods to respond to input information, and there are no limitations on this.

[0174] Reference Figure 10 This is a schematic diagram of the hardware structure of an electronic device proposed in an embodiment of this application. The electronic device may include at least one memory 101 and at least one processor 102, wherein: the processor 102 is used to load a plurality of computer instructions stored in the memory 101, execute the computer instructions, and implement the speech synthesis method proposed in any embodiment of this application. The implementation process can be referred to the description of the corresponding part of the method embodiment above.

[0175] Optionally, when the electronic device is a terminal device, the processor 102 can also send multimodal data to a cloud device, whereby the cloud device calls a deployed speech synthesis model to process the multimodal data to obtain a first feature, infers a second feature from it, and then inputs it into an inference prediction model deployed in the cloud or locally on the terminal device for iterative prediction. All candidate acoustic units obtained from consecutive target iterations are transmitted to the speech synthesis model in the cloud for screening, and the target speech is synthesized using the obtained target acoustic units and then fed back to the terminal device, etc.

[0176] Optionally, the processor 102 may also invoke an intelligent program (such as an intelligent agent) stored in the memory 101 to enter a working state. The intelligent program is configured to implement the speech synthesis method described in the embodiments above, and output the final generated target speech through at least one output component.

[0177] It should be understood that, Figure 10 The structure of the electronic device shown does not constitute a limitation on the electronic device in the embodiments of this application. In practical applications, the electronic device may include more than Figure 10 The application does not provide detailed examples of all of the more or fewer components shown, or combinations of certain components, such as sensing units like gyroscopes, accelerometers, and gravity sensors, power management modules, antennas, or other communication elements.

[0178] Finally, it should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the accompanying drawings of the device embodiments provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0179] In the above embodiments, the invention can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. Through the description of the above embodiments, those skilled in the art can clearly understand that this application can be implemented using software plus necessary general-purpose hardware, or it can be implemented using dedicated hardware including dedicated integrated circuits, dedicated CPUs, dedicated memory, dedicated components, etc. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The various embodiments in this specification are described in a progressive or parallel manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to mutually. For the apparatuses and electronic devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

Claims

1. A speech synthesis method, comprising: Obtain a first feature input from the speech synthesis model, the first feature being obtained based on multimodal data processing; Obtain a second feature generated by the speech synthesis model based on the first feature, wherein the second feature is a speech semantic feature associated with the acoustic attributes of the speech; Based on the first feature and the second feature, iterative prediction is performed through an inference prediction model to obtain at least one candidate acoustic unit corresponding to the output position of each successive target iteration prediction. Based on multiple candidate acoustic units predicted in successive target iterations, the speech synthesis model filters out multiple target acoustic units with consecutively adjacent positions to synthesize target speech. The model parameters of the inference prediction model are smaller than those of the speech synthesis model.

2. The method according to claim 1, wherein, The inference prediction model and the speech synthesis model use the same inference parameters to generate acoustic units, so as to unify the prediction probability distribution of acoustic units by the inference prediction model and the speech synthesis model. The step of iteratively predicting using an inference prediction model to obtain at least one candidate acoustic unit corresponding to the output position of each successive target iteration includes: By performing the nth iteration prediction through the inference prediction model, at least one acoustic unit to be processed is obtained. Based on the candidate acoustic units obtained from the first n-1 iterations of prediction, the acoustic units to be processed are deduplicated to obtain at least one candidate acoustic unit corresponding to the output position of each of the consecutive n iterations of prediction. The output position of at least one candidate acoustic unit predicted in the nth iteration is adjacent to the output position of at least one candidate acoustic unit predicted in the (n-1)th iteration.

3. The method according to claim 2, wherein the iterative prediction using the inference prediction model further comprises: Based on a pre-configured repetition penalty parameter, the prediction probability of the same acoustic unit as each candidate acoustic unit predicted in the previous n-1 iterations is reduced in the nth iteration, so that the nth iteration is based on the adjusted prediction probability. The repetition penalty parameter can be determined based on the quality requirements of the target speech in the speech synthesis scenario.

4. The method according to claim 2, wherein the iterative prediction using the inference prediction model further comprises: Initialize a two-dimensional array to store candidate acoustic units; The candidate acoustic units remaining after each iteration of prediction deduplication are stored in the same row of the two-dimensional array; When the number of iterations reaches the target number, each candidate acoustic unit stored in the two-dimensional array is input into the speech synthesis model for probability verification; Based on the probability verification results, multiple target acoustic units with consecutively adjacent output positions are selected.

5. The method according to any one of claims 1-4, wherein, The second feature is a multi-level feature set with different speech representation granularities generated by the speech synthesis model decoding the first feature layer by layer. The step of iteratively predicting based on the first feature and the second feature using an inference prediction model to obtain at least one candidate acoustic unit at the corresponding output position for each of the consecutive target iterations includes: Based on the first feature and the second feature, the decoded output features are obtained through an inference prediction model; The decoding output features are continuously predicted using the inference prediction model to obtain at least one candidate acoustic unit in each iteration.

6. The method according to claim 5, wherein obtaining the decoded output features based on the first feature and the second feature through an inference prediction model comprises: The first feature is transformed by the inference prediction model to obtain a first intermediate feature that matches the feature dimension of the inference prediction model. The second feature is fused using the inference prediction model to obtain a second intermediate feature that matches the feature dimensions of the inference prediction model; The first intermediate feature and the second intermediate feature are fused and decoded using the inference prediction model to obtain the decoded output feature.

7. The method according to claim 6, wherein, The inference prediction model has the same prediction output module and acoustic unit vocabulary as the speech synthesis model. The prediction output module can output all candidate acoustic units for consecutive target iterations based on the inference parameters and the received decoded output features. The model parameters of the fully connected module for obtaining the first intermediate feature, the mapping processing module for obtaining the second intermediate feature, and the fusion decoding module in the inference prediction model are all trained based on the speech synthesis model.

8. The method according to claim 2, wherein, The inference parameters are determined based on the inference performance requirements in the speech synthesis scenario, and the inference performance requirements include at least one of inference speed and inference diversity. The inference parameters determined based on the inference speed are less than the inference parameters determined based on the inference diversity.

9. The method according to claim 1, wherein the multimodal data includes at least two combinations of video data, audio data, and text data; The first feature includes: Visual hidden features obtained by encoding the video data, audio hidden features obtained by encoding the audio data, and text hidden features and text encoding obtained by encoding the text data; The second feature includes: high-granularity features, medium-granularity features, and low-granularity features; The high-granularity features characterize the acoustic style information of the speech; The medium-granularity features characterize the prosodic temporal information of speech; The low-granularity features represent the basic acoustic pronunciation information of speech.

10. An electronic device comprising at least one memory and at least one processor, wherein: The processor is configured to load a plurality of computer instructions stored in the memory, execute the computer instructions, and perform the following steps: Obtain a first feature input from the speech synthesis model, the first feature being obtained based on multimodal data processing; Obtain a second feature generated by the speech synthesis model based on the first feature, wherein the second feature is a speech semantic feature associated with the acoustic attributes of the speech; Based on the first feature and the second feature, iterative prediction is performed through an inference prediction model to obtain at least one candidate acoustic unit at the output position corresponding to each of the consecutive target iterations. Based on multiple candidate acoustic units predicted in successive target iterations, the speech synthesis model filters out multiple target acoustic units whose output positions are continuously adjacent in order to synthesize target speech.

Images