Voice generation method and device based on pseudo-autoregression modeling, equipment and medium

Abstract

The invention relates to the technical field of voice semantics, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a voice generation method, device and equipment based on pseudo-autoregression modeling and a medium, and the method comprises the steps: obtaining a training sample containing a text sequence, a prompt voice segment and a target semantic token sequence; performing continuous fragment mask training on the text-to-semantic model to obtain a pseudo-autoregression trained text-to-semantic model; generating candidate speech output by using the text-to-semantic model and the initial semantic-to-acoustic model which are subjected to pseudo-autoregression training, and constructing a preference data pair; updating the semantics-to-acoustics model based on the preference data pair to obtain a preference optimized semantics-to-acoustics model; and generating target voice output based on the target text and the target prompt voice. According to the method, the time sequence modeling capability of the model is enhanced through pseudo-autoregression training, and the voice generation quality is directly optimized through the preference data pair, so that the voice alignment precision and the subjective listening feeling performance are improved.

Description

Technical Field

[0001] This invention relates to the field of speech semantics technology, and in particular to a speech generation method, apparatus, device and medium based on pseudo-autoregressive modeling. Background Technology

[0002] In recent years, the development of large-scale neural codec language models has significantly promoted the progress of zero-sample text-to-speech technology, enabling systems to quickly synthesize natural-sounding speech in the absence of specific speaker data.

[0003] In the fintech business sector, systems typically need to generate clear, stable, and easily understandable speech for customer service, compliance review, and voice-interactive Q&A scenarios to support service processes such as intelligent agents and intelligent advisors. However, existing models suffer from insufficient alignment accuracy and high word error rates in complex business contexts and scenarios with numerous technical terms, often resulting in inconsistencies between the spoken content and the original text, thus affecting the reliability and availability of the business system.

[0004] In the healthcare sector, applications such as intelligent consultation systems, assisted rehabilitation equipment, and health consultation robots require models to maintain a stable rhythm, clear expression, and controllable prosody when faced with inputs of different speaking styles or time scales; otherwise, misunderstandings or negative impacts on the interactive experience may occur. Therefore, these fields place higher demands on the accuracy, stability, and controllability of speech generation.

[0005] Existing text-to-speech technologies mainly fall into two categories: autoregressive models and non-autoregressive models. Autoregressive models rely on a token-by-token sequential generation method. Although they have strong temporal dependency modeling capabilities, their inference speed is slow, making it difficult to meet the latency requirements of real-time scenarios. Furthermore, they are prone to exposure biases during the inference phase that are inconsistent with those during the training phase, further reducing the accuracy of the generated speech. Summary of the Invention

[0006] The main objective of this invention is to provide a speech generation method, apparatus, device, and storage medium based on pseudo-autoregressive modeling. This invention aims to solve the technical problems of existing technologies being unable to achieve stable temporal alignment capabilities while maintaining high-speed non-autoregressive inference, and lacking a speech quality optimization mechanism directly oriented towards human preferences, resulting in high word error rates and limited speech quality.

[0007] To achieve the above objectives, this invention provides a speech generation method based on pseudo-autoregressive modeling, comprising: Obtain training samples containing text sequences, prompt speech fragments, and target semantic token sequences; The text-to-semantic model is trained based on the training samples. During training, a continuous segment masking strategy is applied to the target semantic token sequence, and the text-to-semantic model is trained to predict only the leftmost segment of the masked part, resulting in a text-to-semantic model trained by pseudo-autoregression. The training samples are processed using the pseudo-autoregressive text-to-semantic model and the initial semantic-to-acoustic model to generate multiple candidate speech outputs. Based on the word error rate and similarity index of the candidate speech output, the candidate speech outputs are sorted to construct preference data pairs; Based on the preference data pairs, the parameters of the initial semantic-to-acoustic model are updated using the direct preference optimization module to obtain a preference-optimized semantic-to-acoustic model. The target text and target prompt speech are acquired, and then processed by the text-to-semantic model trained by pseudo-autoregression and the semantic-to-acoustic model optimized by preference to obtain the target speech output.

[0008] Furthermore, to achieve the above objectives, the present invention provides a speech generation device based on pseudo-autoregressive modeling, comprising: The training sample construction module is used to obtain training samples containing text sequences, prompt speech fragments, and target semantic token sequences; The text-to-semantic training module is used to train the text-to-semantic model based on the training samples. During training, a continuous segment masking strategy is applied to the target semantic token sequence, and the text-to-semantic model is trained to predict only the leftmost segment of the masked part, thus obtaining a text-to-semantic model trained by pseudo-autoregression. The candidate speech generation module is used to process the training samples using the text-to-semantic model trained by pseudo-autoregression and the initial semantic-to-acoustic model to generate multiple candidate speech outputs. The preference data construction module is used to sort the candidate speech outputs based on the word error rate and similarity index, and construct preference data pairs. The semantic-to-acoustic optimization module is used to update the parameters of the initial semantic-to-acoustic model based on the preference data pair using the direct preference optimization module, so as to obtain a preference-optimized semantic-to-acoustic model. The speech generation module is used to acquire target text and target prompt speech, and process the target text and target prompt speech through the text-to-semantic model trained by pseudo-autoregression and the semantic-to-acoustic model optimized by preference to obtain the target speech output.

[0009] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a speech generation program based on pseudo-autoregressive modeling stored in the memory and executable on the processor, wherein when the speech generation program based on pseudo-autoregressive modeling is executed by the processor, it implements the steps of the speech generation method based on pseudo-autoregressive modeling as described above.

[0010] Furthermore, to achieve the above objectives, the present invention also provides a non-volatile computer-readable storage medium storing a speech generation program based on pseudo-autoregressive modeling, wherein when the speech generation program based on pseudo-autoregressive modeling is executed by a processor, it implements the steps of the speech generation method based on pseudo-autoregressive modeling as described above.

[0011] Beneficial Effects: This invention relates to the field of speech and semantic technology and can be applied to business scenarios such as fintech and healthcare. It discloses a speech generation method, apparatus, device, and medium based on pseudo-autoregressive modeling, comprising: acquiring training samples containing a text sequence, prompt speech segments, and a target semantic token sequence; performing continuous segment masking training on the text-to-semantic model to obtain a pseudo-autoregressive-trained text-to-semantic model; generating candidate speech outputs and constructing preference data pairs using the pseudo-autoregressive-trained text-to-semantic model and an initial semantic-to-acoustic model; updating the semantic-to-acoustic model based on the preference data pairs to obtain a preference-optimized semantic-to-acoustic model; and generating target speech output based on the target text and target prompt speech. This invention enhances the temporal modeling capability of the text-to-semantic model through pseudo-autoregressive training and directly optimizes the output of the semantic-to-acoustic model through preference data pairs, thereby improving the generated speech in terms of alignment accuracy, word error rate, and auditory quality. Attached Figure Description

[0012] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a speech generation method based on pseudo-autoregressive modeling in one embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the speech generation method based on pseudo-autoregressive modeling of the present invention. Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the speech generation device based on pseudo-autoregressive modeling of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0013] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0014] The speech generation method based on pseudo-autoregressive modeling provided in this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can obtain training samples containing text sequences, prompt speech segments, and target semantic token sequences from the client; perform continuous segment masking training on the text-to-semantic model to obtain a pseudo-autoregressive trained text-to-semantic model; generate candidate speech outputs and construct preference data pairs using the pseudo-autoregressive trained text-to-semantic model and the initial semantic-to-acoustic model; update the semantic-to-acoustic model based on the preference data pairs to obtain a preference-optimized semantic-to-acoustic model; and generate target speech output based on the target text and target prompt speech. This invention enhances the temporal modeling capability of the text-to-semantic model through pseudo-autoregressive training and directly optimizes the output of the semantic-to-acoustic model through preference data pairs, thereby improving the generated speech in terms of alignment accuracy, word error rate, and auditory quality. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0015] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the speech generation method based on pseudo-autoregressive modeling provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0016] like Figure 2 As shown, the speech generation method based on pseudo-autoregressive modeling proposed in this invention includes the following steps: S10, Obtain training samples containing text sequences, prompt speech segments, and target semantic token sequences; In this embodiment, the text sequence originates from a language corpus, which can be various structured or semi-structured texts such as news texts, intelligent customer service dialogue texts, medical consultation records, and financial business Q&A records. To facilitate subsequent model processing, the text sequence is generally divided into word units that are easier to express semantics using a word segmentation algorithm. The word segmentation algorithm can adopt a dictionary-based segmentation method, or it can use a statistical probability model or a neural network model to perform data-driven word boundary determination. The segmented text sequence serves as a semantic conditional input during training, enabling the model to form a clear semantic expression of the language content at the text level.

[0017] The prompt speech segment originates from the original speech signal of the target speaker, typically obtained through a speech acquisition device. To enhance the model's adaptability to different acoustic conditions, the original speech signal is generally preprocessed to make it more suitable for input into the acoustic model, including noise reduction, pre-emphasis, and endpoint detection. Subsequently, segmentation is performed, dividing the continuous speech into several segments so that the model can extract features such as speaker timbre, speech rate, and prosody from a limited speech context. The prompt speech segment is then converted into a vector sequence by an acoustic feature extractor. This feature extractor can be based on Mel-spectrum analysis, acoustic convolutional encoders, or self-supervised acoustic models, and can extract features expressing prosody, rhythm, and timbre from the speech waveform.

[0018] The target semantic token sequence is a crucial component of the training samples, generated from cue speech segments or other semantic reference data using a semantic token generation model. The semantic token sequence consists of a series of discrete tokens that express the semantic content and prosodic structure corresponding to the speech, serving as an intermediate semantic representation connecting text and acoustic space. The target semantic token sequence is typically generated by a semantic encoder model, which can employ either a sequence encoding structure based on vector quantization or a semantic modeling structure based on mask prediction to describe the semantic alignment between text meaning and speech expression.

[0019] When constructing training samples, it is necessary to match the segmented text sequence, the prompt speech fragment, and the target semantic token sequence to ensure temporal and semantic consistency within the same training sample. Matching methods can be based on alignment metadata or on correspondences generated by automatic text-to-speech alignment tools. The final form of the training samples is typically represented as structured data, combining the three input sources as data input for model training, enabling the subsequent training process to simultaneously learn semantic content, speaker timbre, and semantic token structure.

[0020] This embodiment introduces text sequences, prompt speech segments, and target semantic token sequences, enabling the training input to simultaneously cover textual semantic features, acoustic prosodic features, and intermediate semantic features. The text sequence enhances language understanding, the prompt speech segments strengthen timbre and prosodic expression, and the target semantic token sequence reinforces semantic structure modeling, thereby forming a more stable training signal that provides higher alignment accuracy and stronger generalization ability for subsequent semantic prediction and acoustic generation.

[0021] S20, the text-to-semantic model is trained based on the training samples. During training, a continuous segment masking strategy is applied to the target semantic token sequence, and the text-to-semantic model is trained to predict only the leftmost segment of the masked part, thus obtaining a text-to-semantic model trained by pseudo-autoregression. In this embodiment, the process of training the text-to-semantic model based on training samples relies on the joint input of a text sequence, a prompt speech segment, and a target semantic token sequence. The text sequence is responsible for expressing semantic content, and a sequence that can represent the language structure is obtained through character segmentation or sub-word modeling. The prompt speech segment provides acoustic characteristics such as timbre, prosody, and speech rate, providing a reference for the model to learn the implicit mapping between semantics and acoustics. The target semantic token sequence records the correspondence between text semantics and speech structure in the form of discrete tokens, and serves as a supervision signal for semantic prediction during model training.

[0022] During training, the target semantic token sequence needs to be masked using a continuous segment masking strategy. This strategy masks semantic tokens over time by randomly selecting a token index as the starting position and hiding all tokens from that index to the end of the sequence, preventing the model from directly observing the masked portion. This allows the model to infer the content of the masked part from the visible part when faced with missing semantic structures during training. The mask length can cover short semantic segments or multiple semantic segments, enabling the model to learn the association between the text sequence and the accompanying speech segments within different coverage areas, achieving stronger temporal sequence modeling capabilities.

[0023] When training the text-to-semantic model, only the leftmost segment of the masked portion is predicted. The leftmost segment refers to a single semantic token at the beginning of a series of masked segments. This prediction method forces the model to infer the earliest semantic structure from the text sequence and the accompanying audio clips, thus constructing a pseudo-autoregressive generation pattern. The model needs to predict based solely on the unmasked preceding semantic tokens, the text sequence, and the accompanying audio clips, without any future semantic tokens, to simulate temporal causality, thus forming a pseudo-autoregressive training mechanism. When predicting the leftmost segment, the model performs forward propagation, outputting a probability distribution containing all possible tokens, and obtains the error signal by comparing the predicted values ​​with the actual tokens.

[0024] After an error occurs, the model's internal parameters need to be adjusted through backpropagation to make the prediction result closer to the target semantic token sequence. Backpropagation calculates gradients, propagating the error layer by layer along the network structure and updating the model parameters according to the gradient direction, gradually enabling the model to distinguish between different semantic and acoustic conditions. The training process involves repeatedly executing mask generation, data input, leftmost segment prediction, error calculation, and parameter updates until termination conditions are met, such as loss stabilization or the number of training epochs reaching a set threshold, thus obtaining a text-to-semantic model trained using pseudo-autoregression.

[0025] This embodiment combines continuous segment mask training with leftmost segment prediction to enable the text-to-semantic model to possess pseudo-autoregressive temporal causality, enhancing the model's inference ability under missing semantic conditions. This training method improves the stability and continuity of semantic prediction, allowing the model to better align text sequences with semantic token structures, thereby improving the quality of semantic generation and temporal structure consistency.

[0026] S30, the training samples are processed using the pseudo-autoregressive text-to-semantic model and the initial semantic-to-acoustic model to generate multiple candidate speech outputs; In this embodiment, when processing training samples using a text-to-semantic model trained with pseudo-autoregression, it is necessary to receive text sequences and prompt speech segments, mapping both types of data together into a semantic token space. The text sequence records language content and structure, while the prompt speech segments carry acoustic information, such as timbre, rhythm, and prosody. The combined input of both enables the semantic prediction process to infer based on both language expression and acoustic conditions. The text-to-semantic model trained with pseudo-autoregression possesses temporal causality, allowing it to reference prior semantic relationships when predicting semantic output, thus ensuring that the generated semantic tokens have a consistent temporal structure. The model encodes the input text sequence and prompt speech segments, fusing text and speech features into a joint representation, and inferring the semantic token sequence from it, giving the semantic representation a clear temporal progression relationship.

[0027] To generate multiple candidate speech outputs, the generated semantic token sequence and the prompt speech fragment need to be input into an initial semantic-to-acoustic model. The initial semantic-to-acoustic model converts the semantic token sequence into an acoustic token sequence, outputting acoustic units representing speech content by learning the mapping relationship between semantic tokens and acoustic representations. The parameters of this model reflect the initial mapping ability from semantic information to specific acoustic expressions and have not yet been optimized for preferences; therefore, the generated acoustic token sequences exhibit a certain degree of diversity.

[0028] To obtain multiple candidate acoustic token sequences, a random seed needs to be introduced during the acoustic generation process. The random seed controls the sampling behavior during model decoding, ensuring that different random seeds guide the semantics to the acoustic model to generate different acoustic paths while maintaining semantic consistency. These paths may include different prosody choices, pause positions, or subtle timbre differences. The random seed setting needs to participate in probability distribution sampling during decoding to ensure that each acoustic generation has independent randomness.

[0029] After the acoustic token sequence is generated, it needs to be input into a vocoder to map the discrete acoustic tokens to a continuous audio waveform. The vocoder learns the correspondence between acoustic tokens and real speech waveforms during the acoustic training phase, thus enabling it to reconstruct a playable speech signal from the acoustic token sequence. To obtain multiple candidate speech outputs, the random seed setting, acoustic token generation, and audio synthesis processes need to be repeated, ensuring that each generated speech signal exhibits different acoustic characteristics under the same semantic conditions.

[0030] This embodiment combines text-to-semantic generation, randomized acoustic generation, and vocoder reconstruction to enable training samples to generate multiple semantically consistent but acoustically different candidate speech outputs. This provides the necessary diversity for subsequent preference ranking, thereby constructing more accurate preference data pairs and improving the effectiveness of subsequent model optimization.

[0031] S40, sort the candidate speech outputs based on word error rate and similarity index, and construct preference data pairs; In this embodiment, given that multiple candidate speech outputs have been generated, quality assessment and preference differentiation are performed using two metrics: word error rate (BER) and similarity. The candidate speech outputs are multiple speech generation results generated from the same text sequence and prompt speech fragments. To measure the deviation between these outputs and the text semantics, each candidate speech output needs to be input into an automatic speech recognition system, which transcribes the audio signal into a text sequence. The BER can be calculated by comparing the recognized text with the text sequences in the training samples. The BER is typically calculated based on edit distance, measuring the degree of deviation between the spoken expression and the text content through the number of replacements, insertions, and deletions. The text sequences originate from the input text generated during the training phase and can therefore be directly used for comparison.

[0032] On the other hand, the differences in sound features among candidate speech outputs need to be evaluated using a similarity metric. To this end, it is necessary to extract the acoustic feature vectors of both the candidate speech outputs and the corresponding prompt speech segments. These acoustic feature vectors, derived from Mel-spectrum analysis, f0 features, energy curves, or multidimensional speech embeddings, can provide a multidimensional description of sound timbre, prosody, and resonance structure. By calculating the cosine similarity between the two vectors, the degree of closeness between the candidate speech outputs and the prompt speech segments in terms of timbre consistency can be characterized; a larger cosine value indicates a closer similarity in sound characteristics.

[0033] To ensure that word error rate and similarity metrics are used together in the ranking, both metrics need to be normalized to ensure their range and dimensions are consistent. Normalizing word error rate strengthens effective comparisons, while normalizing cosine similarity compresses excessively dispersed value ranges, making the scoring system more stable. Subsequently, the normalized word error rate score and similarity score need to be weighted and summed to obtain a comprehensive score. The weights can be adjusted according to the focus of the speech task; for example, increasing the weight of word error rate when emphasizing clarity, and increasing the weight of similarity when emphasizing timbre consistency.

[0034] After the overall score is generated, all candidate speech outputs need to be sorted from highest to lowest score. The purpose of sorting is to select the set of data that best meets the quality requirements from multiple candidate outputs to construct preference data pairs. The highest-scoring outputs have the best text consistency and timbre consistency, and therefore can be used as preference samples; the lowest-scoring outputs reflect the worst overall performance and can be used as non-preference samples. Finally, preference samples and non-preference samples need to be combined to form data pairs for supervised preference optimization training.

[0035] This embodiment ranks candidate speech outputs using two metrics: word error rate and similarity, and constructs preference data pairs. This enables the training phase to obtain optimized signals that simultaneously consider text consistency and timbre consistency, thereby improving the model's ability to respond to human subjective preferences and achieving a more natural and stable speech generation effect.

[0036] S50, based on the preference data pair, the parameters of the initial semantic-to-acoustic model are updated using the direct preference optimization module to obtain a preference-optimized semantic-to-acoustic model; In this embodiment, based on the already constructed preference data pairs, the initial semantic-to-acoustic model needs to be updated with parameters so that the model can adjust its output along the preference direction. The preference data pairs consist of preferred and non-preferred samples; each pair is generated based on the same input conditions, thus allowing direct comparison of the model's quality differences between different outputs. To enable the model to learn these differences, a combination of input conditions, preferred samples, and non-preferred samples needs to be input into the optimization process.

[0037] First, the initial semantics need to be loaded into the acoustic model and used as the policy model to be updated. The policy model is responsible for calculating the output probabilities to evaluate the model's output tendency on preferred and unpreferred samples. To provide a stable optimization reference, a static copy of the policy model needs to be created, and all parameters of the copy are frozen, making it the reference model. The output of the reference model is used to measure the difference between the policy model and the original parameter state, thus enabling the optimization process to maintain the balance of the probability distribution.

[0038] Subsequently, log probabilities need to be calculated for each training batch. Specifically, the input conditions and preferred samples are input into the policy model to obtain the log probability of the policy model with respect to the preferred samples; the same input conditions and non-preferred samples are input into the policy model to obtain the log probability of the policy model with respect to the non-preferred samples. The reference model calculates the corresponding log probabilities for the same two sets of inputs respectively, forming a matching reference value. The calculation of log probabilities is used to measure the probability strength of the model generating a certain output under given input conditions, and can directly reflect the probability difference between preferred and non-preferred samples.

[0039] Subsequently, the log probability difference between the policy model and the reference model is processed. Two differences are constructed: the difference in preferred samples and the difference in unpreferred samples, to describe the bias of the model towards the two types of samples. The difference in preferred samples reflects the improvement of the policy model in the preferred direction compared to the reference model, while the difference in unpreferred samples reflects the degree of degradation of the policy model in the undesired direction.

[0040] After obtaining the two differences, these differences need to be input into the direct preference optimization loss function. The loss function compares the probability differences between preferred and non-preferred samples, causing the policy model to favor increasing the probability of preferred sample outputs while suppressing the probability of non-preferred sample outputs. The loss function is constructed based on the relative value of the log probability difference, enabling the model to directly learn preference ranking relationships rather than learning absolute distributions.

[0041] After the loss value is generated, the gradient of the loss relative to the policy model parameters is calculated to form an update signal in the parameter direction. Gradient calculation relies on the backpropagation mechanism, which transmits preference differences to the model parameters through the computation graph, causing the parameters to adjust in a direction that better aligns with the preference ranking. The parameters of the policy model are updated based on the gradient, completing one training iteration.

[0042] The entire process needs to be repeated to allow the policy model to gradually approach the desired preference ranking capability. After training, the policy model, which has undergone multiple rounds of preference updates, becomes a preference-optimized semantic-to-acoustic model with higher subjective quality output capability.

[0043] This embodiment updates the parameters of the initial semantics to the acoustic model under the drive of preference data pairs, enabling the model to directly learn preference ranking relationships, thereby strengthening its sensitivity to subjective quality and improving the overall performance of the generated speech in terms of clarity, naturalness, and timbre consistency.

[0044] S60, acquire the target text and target prompt speech, process the target text and target prompt speech through the text-to-semantic model trained by pseudo-autoregression and the semantic-to-acoustic model optimized by preference, and obtain the target speech output.

[0045] In this embodiment, the target text and target prompt speech need to be acquired first. The target text refers to the character sequence used to describe the desired generated speech content, and can originate from typed content, interface input, or text generated by upstream tasks. The text is typically processed through character segmentation, word mapping, or embedding transformations to ensure compatibility with the semantic generation mechanism. The target prompt speech refers to an audio segment serving as a style reference or timbre guide, and can include natural human voice, character-style voice, or short recordings captured by the device. The prompt speech is typically normalized, silenced, or feature-extracted to ensure it shares the same acoustic space as the semantic generation process.

[0046] After acquiring the input, the target text and the target prompt speech input need to be processed by a text-to-semantic model trained using pseudo-autoregression. The text-to-semantic model has already established temporal causality through pseudo-autoregression training, thus enabling it to generate semantic token sequences based on the acoustic features of the prompt speech and the semantic structure of the text. The semantic token sequence is an intermediate representation between the text and acoustic space, typically composed of discrete vector indices, used to describe the semantic skeleton of the speech content, including pronunciation patterns, prosodic segments, and sentence rhythmic structure. The pseudo-autoregressive mechanism ensures these tokens possess stable temporal order properties through continuous segment prediction.

[0047] The generated semantic token sequence is then compared with the target prompt speech input to a preference-optimized semantic-to-acoustic model. The semantic-to-acoustic model learns preferences for tone quality, intelligibility, and stability, and can generate an acoustic token sequence based on the semantic token sequence. The acoustic token sequence is used to construct the final discrete features of the speech waveform, typically containing information such as pitch, energy, formants, and rhythm. To maintain timbre consistency, features of the target prompt speech are also incorporated, ensuring that the generated speech retains the language content while possessing a reference timbre.

[0048] After obtaining the acoustic token sequence, it needs to be input into a vocoder. The vocoder converts the token sequence into a playable audio waveform through continuous inverse transform, parameter mapping, or spectral reconstruction mechanisms. The vocoder can be implemented based on a generative network, phase reconstruction algorithm, or hybrid spectral model, and converts the discrete acoustic representation into a continuous speech signal through sample-by-sample reconstruction or frame-by-frame reconstruction. After amplitude adjustment, post-processing, or sampling rate conversion, the waveform signal becomes the target speech output.

[0049] The final output target speech must maintain the same content as the input target text, retain the timbre trend of the input target prompt speech, and possess the subjective quality advantages formed by the preference-optimized model.

[0050] This embodiment uses a hierarchical structure of text-to-semantic generation and semantic-to-acoustic generation, combined with prompting speech and preference learning mechanisms, to achieve a balance between content accuracy, timbre consistency and subjective sound quality in the final output speech, thereby obtaining stable and high-quality speech generation results in multiple scenarios.

[0051] In one embodiment, step S10 above includes: S101, Obtain the original text data, perform word segmentation on the original text data, and obtain the word-segmented text sequence; S102, Obtain the original prompt voice, preprocess and segment the original prompt voice to obtain the preprocessed prompt voice segment; S103, The preprocessed prompt speech segment is processed using an acoustic feature extractor to obtain an acoustic feature sequence; S104, input the segmented text sequence and the acoustic feature sequence into the pre-trained semantic token generation model to obtain the target semantic token sequence; S105, combine the segmented text sequence, the preprocessed prompt speech segment, and the target semantic token sequence to obtain training samples.

[0052] In this embodiment, the stability and semantic integrity of the source are prioritized when acquiring raw text data. The text cleaning process includes strategies such as character set unification, full-width / half-width character conversion, abnormal symbol replacement, whitespace folding, and punctuation preservation to retain necessary pause information during subsequent speech alignment. Language tags and domain labels can be saved as additional identifiers along with the text for controlling speech rate and tone in later stages. Text quality control reduces the impact of extreme samples through sentence length distribution checks and duplicate detection, and maintains uniqueness using hashing or fingerprinting methods to prevent the same data from being counted repeatedly during training.

[0053] Word segmentation maps the original text data into a segmented text sequence. Two approaches can be used: sub-word unit construction and joint word-character modeling. Sub-word unit construction emphasizes transferability, establishing a vocabulary based on statistical co-occurrence and segmentability principles, and optimizing the boundary with the goal of maximizing coverage and minimizing unknown units. Joint word-character modeling preserves the integrity of high-frequency words while providing sub-word segmentation for low-frequency long words, balancing expressiveness and robustness. Regularization is performed during word segmentation to map synonyms to unified tags, and case folding and number placeholder strategies are used to reduce sparsity caused by surface variations. The segmented text sequence is stored as a fixed-length or variable-length sequence, with accompanying masks and attention boundaries to accommodate subsequent parallel or autoregressive learning mechanisms.

[0054] When acquiring the original prompt speech, it is necessary to ensure that the sampling rate, number of channels, and encoding format are consistent to facilitate batch processing and feature isomorphism. Recording segments are screened for duration, peak pruning, and loudness consistency to eliminate recording differences. Environmental noise and reverberation are reduced through noise reduction and reverberation suppression strategies, and silent intervals are determined jointly by energy threshold and speech activity detection. Segment segmentation follows a trade-off between the smallest semantic unit and the largest prosodic span. If text alignment information exists, segmentation is prioritized according to sentence boundaries; otherwise, speech activity boundaries are used. For cross-boundary short sentences, slight overlap is used to preserve contextual timbre clues. After processing, preprocessed prompt speech segments are formed, and start and end timestamps and segment identifiers are recorded to ensure traceability with the text side.

[0055] The preprocessed prompt speech segments are processed using an acoustic feature extractor to obtain an acoustic feature sequence. Frame-level processing generates a time-frequency representation through a sliding window, commonly using log-Mel energy or cepstral coefficients to represent the spectral envelope, combined with first- and second-order dynamics to capture short-term variations; latent vectors from the output of a self-supervised acoustic encoder can also be introduced to enhance robustness. Feature normalization reduces recording differences through conversation-level or speaker-level mean-variance standardization, and inter-channel alignment ensures consistency in time and amplitude across multiple microphone inputs. To preserve timbre and prosody, the feature pipeline retains fundamental frequency, energy, and persistence information channels, and improves stability through missing frame imputation and aberration repair. The final output acoustic feature sequence is one-to-one associated with the segment identifier, providing a basis for cross-modal alignment and downstream modeling.

[0056] The segmented text sequence and acoustic feature sequence are input into a pre-trained semantic token generation model to obtain the target semantic token sequence. The semantic token generation model bridges the semantic representation to abstract speech expression, and common implementations include discretized autoencoder structures or quantization bottleneck structures. On the text side, language representations are constructed through embedding and positional encoding; on the speech side, acoustic context representations are formed through temporal compression and cross-channel fusion. Cross-modal alignment is achieved through interactive attention or gated fusion, aligning text prosodic markers with the timbre cues of the prompt speech in the same representation space. The quantization layer forms discrete tokens using codebook indexes, and perceptual loss constraints ensure semantic fidelity. The training process, which carries mask reconstruction or contrast constraints, improves temporal consistency and composability. The target semantic token sequence output is a stable discrete index stream, maintaining its mapping relationship with text markers and the acoustic timeline, facilitating reuse in subsequent pseudo-autoregressive training and inference stages.

[0057] The segmented text sequence, preprocessed prompt speech segments, and target semantic token sequence are combined to form training samples. The combination strategy uses a unified sample container to package three types of data and associated metadata, including the length of the segmented text sequence, segment boundaries, language or domain labels, the start and end times, feature lengths, and recording environment labels of the preprocessed prompt speech segments, and the index stream and time alignment mapping of the target semantic token sequence. During the batch processing and packaging stage, padding and attention mask generation are performed to ensure effective computation of samples of different lengths within the same batch; simultaneously, a consistent random seed and data augmentation tags are generated to reproduce semantically consistent randomness when generating multiple candidate speech outputs.

[0058] This embodiment establishes strict consistency and traceability at the text, prompt speech, and discrete semantic levels. The segmented text sequence ensures the stability of language content expression, the preprocessed prompt speech segments provide timbre and prosody priors, the acoustic feature sequence standardizes recording differences in the parameter space, and the pre-trained semantic token generation model compresses cross-modal information into target semantic token sequences while maintaining temporal mapping. Finally, training samples are output in a consistent data container. This data organization allows subsequent pseudo-autoregressive training to directly inherit explicit temporal inductive biases and provides highly aligned input-output pairs for the preference learning stage, thereby achieving a stable overall gain between word error rate, naturalness, and timbre consistency.

[0059] In one embodiment, step S20 above includes: S201, For the target semantic token sequence, randomly generate the mask start position; S202, Starting from the mask start position, mask the continuous segments in the target semantic token sequence up to the end of the sequence to generate the masked semantic token sequence; S203, combine the text sequence, the prompting voice segment, and the masked semantic token sequence into model input data; S204, Input the model input data into the text into the semantic model, perform forward propagation processing, and obtain the prediction result of the mask position token; S205, extract the predicted value corresponding to the leftmost segment of the masked continuous segment from the prediction result; S206, determine the loss value between the predicted value of the leftmost segment and the real token in the target semantic token sequence; S207, Based on the loss value, update the parameters of the text-to-semantic model through the backpropagation module until the preset training termination condition is reached, and obtain the text-to-semantic model trained by pseudo-autoregression.

[0060] In this embodiment, the target semantic token sequence refers to a discrete index stream aligned with the text semantics and the prosodic speech of the prompts. It originates from the encoding output of the previous discretizer or quantization bottleneck, maintaining a monotonic mapping to the time axis to perform time-dependent masking operations. Randomly generated mask starting positions employ uniform sampling or length-weighted sampling, ensuring that different positions have approximately equal learning opportunities. The sampling results are mapped to the token index space, and intra-batch offsets are recorded to guarantee that mask segments from different samples within the same batch do not interfere with each other. The strategy of masking continuous segments until the end of the sequence replaces all indices after the starting position with masking markers or placeholders without information, equivalent to retaining only the left-hand context, allowing the model to learn temporal causality even with missing future information. The mask length automatically changes with the sample length, avoiding mismatches caused by static lengths. The masked semantic token sequence retains the unmasked left-hand indices and uses a uniform masking marker on the right-hand side, accompanied by a one-dimensional attention mask and valid bit markers, ensuring that attention only takes effect within the visible region during forward propagation.

[0061] In this training, the text sequence and the prompt speech segment serve as conditional inputs, providing semantic and timbre priors. The three inputs are aligned and combined to form the model input data. The combination method is parallel branch encoding and late fusion: the text is encoded into a language representation through embedding and relative position encoding; the prompt speech segment is encoded into timbre and prosody representations through an acoustic projection layer; and the masked semantic token sequence is encoded into a partially visible target representation through a token embedding layer. The three are then fused into a time-aligned latent representation tensor through interactive attention or gated weighting, with an effective bit mask to shield the leakage of masked areas.

[0062] The forward propagation process performs multi-layer attention and feedforward transformation on the fused latent representation, outputting the class distribution for each time position. To match the pseudo-autoregressive paradigm, the supervision signal is computed only for the leftmost segment of the masked continuous segment. The location of the leftmost segment is directly given by the starting position, and its corresponding predicted value is extracted from the model output distribution by index at that position and aligned with the true index of the target semantic token sequence at the same position. The loss value uses classification cross-entropy or its temperature-scaling variant, and label smoothing can be superimposed to alleviate overfitting. To enhance temporal stability, a consistency regularization between adjacent frames can be introduced to constrain the distribution differences between adjacent positions within an acceptable range but not include them in the main objective, so as not to deviate from the training criterion of "predicting only the leftmost segment".

[0063] The backpropagation module backtracks gradients on the computation graph based on the loss value. Gradients are distributed within the three-way branch and fusion layer, and the optimizer updates parameters using momentum or adaptive second-order approximation. To avoid gradient leakage in occluded regions, the mask also applies in the backpropagation, ensuring that invisible regions do not participate in gradient accumulation. Training termination conditions can be determined by upper limits on the number of rounds, early stopping when validation set metrics show no improvement, or smooth convergence of the loss. The resulting text-to-semantic model, trained via pseudo-autoregression, possesses explicit temporal causality and can learn the left-to-right generation bias without relying on autoregressive stepwise decoding. The entire process utilizes random starting points within batches, variable-length alignment across samples, unified occlusion markers, and attention masks to form a stable and repeatable training organization.

[0064] This embodiment uses continuous segment masking up to the end and only supervises the leftmost segment. The model is forced to establish causal dependencies from left to right under the condition of missing future information. Temporal induction bias is injected within the parallel computing framework. Conditional fusion of text and prompt speech aligns semantic selection and timbre prosody in the same latent space. Strict masking and backmasking avoid information leakage and gradient contamination.

[0065] In one embodiment, step S30 above includes: S301, Input the text sequence and the prompt voice segment into the pseudo-autoregressive trained text-to-semantic model to generate a semantic token sequence; S302, Set a random seed, input the semantic token sequence and the prompt speech segment into the initial semantic-to-acoustic model, and generate an acoustic token sequence based on the random seed; S303, synthesize the acoustic token sequence into a candidate speech waveform; S304, Repeat the steps of setting a random seed, generating an acoustic token sequence and synthesizing candidate speech waveforms multiple times to obtain multiple candidate speech waveforms; S305, output the multiple candidate speech waveforms as multiple candidate speech.

[0066] In this embodiment, the text sequence and the prompt speech segment are first aligned temporally and semantically, unifying character specifications, word segmentation boundaries, and sampling rates. A correspondence between text time anchors and speech time anchors is established, eliminating blank segments and anomalous energy segments to ensure that subsequent conditional coding does not introduce cross-modal drift. The text-to-semantic model, trained via pseudo-autoregression, receives the joint input of the text sequence and the prompt speech segment. The text branch generates semantic embeddings, while the prompt speech branch extracts timbre and prosodic cues. These two are aligned and fused along the time axis to output a semantic token sequence. This sequence retains its monotonic relationship with the text semantics and inherits the speaking style and pause rhythm of the prompt speech segment, ensuring stable prior knowledge for subsequent acoustic generation.

[0067] A random seed is used to control the randomness of the acoustic sampling process. The pseudo-random sequence entering the acoustic sampler is initialized by this random seed, affecting the sampling order and candidate distribution extraction of candidate paths. This ensures that differentiated acoustic token sequences are obtained by changing the random seed under the same input conditions, while maintaining complete reproducibility when the random seed is fixed. The initial semantic-to-acoustic model receives the semantic token sequence and prompt speech segments as conditions, constructs frame-level or segment-level probability distributions, and generates acoustic token sequences through a sampling strategy driven by the random seed. The sampling strategy can combine distribution truncation and entropy constraints to suppress excessive diffusion of low-confidence branches and introduce controlled random perturbations in the local context to improve diversity. To avoid semantic rhythm misalignment, constraints are set for duration consistency and prosodic boundaries during sampling. When an out-of-bounds trend occurs, it backs to the nearest high-confidence candidate.

[0068] The acoustic token sequence is restored into candidate speech waveforms by the waveform synthesis unit. The synthesis process first maps discrete tokens to continuous acoustic parameters or latent variables, and then reconstructs them into time-domain signals through a decoder. To ensure consistency in amplitude and loudness, loudness normalization and artifact removal filtering are introduced to remove high-frequency ringing and start / end click noise at the decoding end. The above "setting a random seed - generating an acoustic token sequence - synthesizing candidate speech waveforms" is executed multiple times in a loop mechanism to form multiple candidate speech waveforms. Each loop records the random seed, sampling configuration, and synthesis log, constructing a candidate set that corresponds one-to-one with the text sequence and prompt speech segments, facilitating subsequent quality evaluation and preference construction. To improve throughput, the loop runs in batch processing mode, the semantic token sequence is cached and reused in video memory, and the acoustic generation and synthesis stages are pipelined in parallel to avoid redundant encoding overhead. At the same time, duration pruning and silence thresholds are set to ensure that the outputs of multiple candidate speech are consistent in start / end boundaries and overall duration, avoiding alignment penalties in the subsequent comparison stage.

[0069] This embodiment generates multiple candidate speech outputs that cover pronunciation details, prosodic pauses, and subtle differences in timbre by repeatedly sampling and synthesizing under fixed text sequences and prompt speech segments. The controllable diversity driven by random seeds provides sufficient search space for subsequent quantitative screening based on word error rate and similarity, while maintaining a completely reproducible comparison basis. The joint condition of semantic token sequences and prompt speech segments makes the candidates more stable in terms of semantic alignment and style consistency, thereby improving the data quality and convergence efficiency of subsequent preference learning without increasing the complexity of the preceding modeling.

[0070] In one embodiment, step S40 above includes: S401, input each candidate speech output into the automatic speech recognition module to obtain the speech recognition text; S402, compare the speech recognition text corresponding to each candidate speech output with the corresponding text sequence to determine the word error rate value; S403, extract the acoustic feature vector of each candidate speech output, and extract the acoustic feature vector of the corresponding prompt speech segment for each candidate speech output; S404, Determine the cosine similarity value between the acoustic feature vector of each candidate speech output and the acoustic feature vector of the corresponding prompt speech segment; S405, normalize the word error rate values ​​of all candidate speech outputs to obtain word error rate scores; S406, normalize the cosine similarity values ​​of all candidate speech outputs to obtain similarity scores; S407, the word error rate score and similarity score of each candidate speech output are weighted and summed to obtain a comprehensive score; S408, Sort all candidate speech outputs in descending order of the comprehensive score; S409: Select the candidate speech output with the highest comprehensive score from the ranking results as the preferred sample; S410: Select the candidate speech output with the lowest comprehensive score from the ranking results as the unbiased sample; S411, combine the preferred sample with the non-preferred sample into a preference data pair.

[0071] In this embodiment, after establishing a one-to-one correspondence between candidate speech outputs and text sequences, the speech enters the automatic speech recognition module. The recognition process utilizes a unified pronunciation dictionary, text regularization, and punctuation cleaning rules to avoid offsetting results due to punctuation and capitalization differences. After the speech recognition text is generated, it is aligned with the text sequence. During the comparison process, replacement, insertion, and deletion statistics using edit distance metrics are employed to obtain the word error rate. To eliminate interference from abnormal silences or trailing artifacts, endpoint pruning and duration constraint checks are performed on the candidate speech outputs before comparison to ensure that the comparison interval matches the coverage of the text sequence. To enhance the robustness of the cross-language symbol system, a strategy of homologous word inflection and unified numerical reading is applied to the text sequence and speech recognition text to ensure that the word error rate reflects only content deviations rather than writing differences.

[0072] The similarity metric is calculated based on acoustic feature vectors. Candidate speech outputs are segmented using a Mel-frequency cepstral encoder or a self-supervised encoder, and then masked silence removal and energy thresholding are performed on the time axis to form candidate acoustic feature vectors. The prompt speech segment undergoes the same feature extraction path to obtain a reference acoustic feature vector. To align speaker features with prosodic envelopes, the feature vectors employ a hierarchical structure: lower layers represent timbre and formants, while higher layers represent rhythm and pauses. After hierarchical alignment, cosine similarity values ​​are calculated, and duration matching weights are introduced to suppress artificially high similarity scores caused by length differences. If background noise or device echo is present in the prompt speech segment, frequency band noise suppression and echo cancellation are performed first before feature extraction to prevent similarity scores from being influenced by recording conditions.

[0073] Since the word error rate (BER) and cosine similarity (CSR) values ​​have different distribution ranges, normalization is required to integrate them into a unified scoring space. A reverse mapping is applied to the BER values ​​to reflect that smaller errors result in higher scores, while quantile truncation is introduced to reduce the impact of extreme outliers on the overall score. The CSR values ​​are linearly stretched or standardized to ensure sufficient resolution around the median level. After obtaining the BER and CSR scores, weighting coefficients are configured according to task preferences, and a weighted sum is used to form a comprehensive score. To mitigate the risk of acoustic drift or over-smoothing, soft constraints on duration consistency and pause boundary consistency can be added to the comprehensive score, with adjustable weights to accommodate different languages ​​and speaking styles.

[0074] The ranking process arranges all candidate speech outputs from highest to lowest based on their overall scores. If the overall scores are the same, word error rate scores, similarity scores, and duration deviations are compared sequentially to form a stable decision chain, ensuring the ranking results are unambiguous. To improve processing throughput, the entire evaluation chain is executed in batches, with the automatic speech recognition module and feature extraction running in parallel and sharing an acoustic buffer to reduce redundant decoding. Before ranking, abnormal candidates undergo consistency checks, including excessive peak distortion, continuous plosives, and silences. Samples meeting the rejection criteria are marked as invalid and not included in the ranking set. Finally, the highest overall score is selected as the preferred sample, and the lowest overall score is selected as the non-preferred sample, forming a preferred data pair. To enhance the coverage of preference learning, multiple preference data pairs can be generated by pairing the top high-scoring samples with several low-scoring samples according to pairing rules, without changing the terminology and process. The corresponding candidate identifiers and scoring metadata are recorded for subsequent traceability and reproducibility.

[0075] This embodiment generates word error rate scores and similarity scores from candidate speech outputs under a unified evaluation chain, and aggregates them into a comprehensive score in a weighted manner. This enables the formation of discriminative ranking results while taking into account both content accuracy and style fit. With the help of stable parallel evaluation and anomaly consistency review, the selection of preferred and non-preferred samples is more reliable and reproducible. The constructed preference data provides a high signal-to-noise ratio training signal for subsequent parameter updates, thereby accelerating convergence and improving generation quality and style consistency.

[0076] In one embodiment, step S50 above includes: S501, Load the initial semantics into the acoustic model as a policy model, and copy the parameters of the policy model to establish a reference model; S502, Obtain a training batch containing input conditions, preferred samples, and non-preferred samples from the preference data pair; S503, the input conditions and the preference samples are input into the policy model and the reference model respectively, to obtain the first log probability of the policy model with respect to the preference samples and the third log probability of the reference model with respect to the preference samples respectively; S504, the input conditions and the unpreferenced samples are input into the policy model and the reference model respectively, to obtain the second log probability of the policy model for the unpreferenced samples and the fourth log probability of the reference model for the unpreferenced samples respectively; S505, determine a first difference between the first logarithmic probability and the third logarithmic probability, and a second difference between the second logarithmic probability and the fourth logarithmic probability; S506, Based on the first difference and the second difference, determine the loss value using the direct preference optimization loss function; S507, Based on the loss value, the gradient is determined through the backpropagation module, and the parameters of the policy model are updated based on the gradient; S508 repeatedly performs the operations of acquiring training batches, determining log probabilities, determining differences, and updating parameters until the training stopping condition is met, and uses the updated policy model as the preference-optimized semantic-to-acoustic model.

[0077] In this embodiment, after the preference data pairs enter the direct preference optimization module, the initial semantics are first loaded into the acoustic model in the same computational graph and used as the policy model. Simultaneously, the policy model parameters are copied to construct a reference model, and its gradient propagation is frozen. Both share the acoustic codebook, attention configuration, and acoustic conditional input, ensuring that the probability evaluation is under consistent modeling assumptions. The training batch consists of input conditions, preference samples, and non-preference samples. The input conditions and the two types of samples are fed into the policy model and the reference model respectively, resulting in four log probability estimates. During the calculation, the same feature regularization, the same masking rule, and the same decoding temperature are used to eliminate evaluation bias. Subsequently, the difference between the log probability of the policy model and the log probability of the reference model is calculated in the preference sample branch, and the corresponding difference is calculated in the non-preference sample branch. The two are then aggregated according to a set preference tradeoff to form the direct preference optimization loss. This loss depends only on the relative tendency of the two models on the two types of outputs, does not require an external scorer, and can directly drive the policy model parameters towards preference samples and away from non-preference samples.

[0078] To improve stability, log-probability temperature scaling is enabled during optimization to control gradient magnitude, and gradient pruning and parameter norm constraints are used to suppress the risk of explosion. The reference model remains unchanged, while the policy model calculates the gradient and updates parameters based on the direction of loss through the backpropagation module after each batch. To avoid data sampling bias, preference data pairs are balanced and shuffled within each batch, and a brief coverage metric is introduced to monitor the balance of distribution across different speaking styles, rates, and prosodic types. When the monitoring metrics show that preference samples are excessively concentrated on a few speaking conditions, the sampling ratio is dynamically adjusted to restore coverage. To enhance alignment and intelligibility, the policy model uses the same normalization and masking strategies for acoustic conditions during the forward computation phase as during the training phase, ensuring that the log-probability evaluation strictly corresponds to the same generative hypothesis. The early stopping condition is determined by the preference consistency ratio and word error rate changes on the validation set. Stopping is triggered when preference consistency no longer improves and the word error rate regresses. After the stopping condition is met, the updated policy model is used as the preference-optimized semantic-to-acoustic model for subsequent inference.

[0079] This embodiment introduces direct preference optimization based on log probability difference between the policy model and the reference model, which amplifies the discrimination between preferred and non-preferred samples in the training signal, and makes the parameter update direction closer to human ear preference. Combined with temperature scaling, gradient pruning and equalization sampling, the optimization link reduces the risk of overfitting and mode collapse while maintaining convergence stability, thereby improving the intelligibility, style fit and robustness of the generated speech without introducing an external scorer.

[0080] In one embodiment, step S60 above includes: S601, acquire the target text and the target prompt speech, input the target text and the target prompt speech into the semantic model trained by pseudo-autoregression, and generate a semantic token sequence; S602, input the semantic token sequence and the target prompt speech into the preference-optimized semantic-to-acoustic model to generate an acoustic token sequence; S603, the acoustic token sequence is input into the vocoder, and the vocoder converts the acoustic token sequence into a speech waveform signal to obtain the target speech output.

[0081] In this embodiment, the target text and target prompt speech undergo format unification and security filtering before entering the inference process. On the text side, invalid control characters are removed and lexical mapping is performed, while capitalization, punctuation, and stop words are retained to maintain semantic rhythm. On the speech side, energy normalization and silence segment pruning are performed, while fundamental frequency and formant information are retained to carry timbre and speaking style. The target text and target prompt speech are simultaneously fed into a text-to-semantic model trained by pseudo-autoregression. The text embedding representation and the prompt speech conditional representation are aligned in a multi-branch fusion unit. The time axis establishes alignment anchors based on the text sequence position, and the prompt speech condition provides constraints on speaker timbre, speech rate, and prosody in the channel dimension. During inference, the model reuses the continuous segment mask inductive bias from the training phase, maintaining only the "leftmost segment priority" generation order constraint internally while maintaining a one-time parallel output externally to generate a semantic token sequence. To avoid rhythmic collapse at sentence pauses, the model adds lightweight context gating at semantic boundaries, so that the semantic token sequence forms distinguishable boundary markers both within and between sentences.

[0082] The semantic token sequence and the target prompt speech are then input into a preference-optimized semantic-to-acoustic model. The semantic side channel transmits word-level and phrase-level prosodic markers, while the prompt speech conditional channel provides speaker embeddings and prosodic contours. These two elements establish a one-to-one cross-modal constraint in the conditional fusion unit, generating the acoustic token sequence. To suppress articulation jitter and plosive diffusion, the inference phase maintains the same temperature and sampling strategy as during training, and phase continuity constraints and transition window functions are used at candidate acoustic segment splicing points to ensure the continuity of formant trajectories and energy envelopes. The acoustic token sequence is fed into a vocoder to complete waveform reconstruction. The vocoder's receiving conditions include Mel-spectral approximation and acoustic token time indexes, outputting single-channel or multi-channel speech waveforms. To reduce tail trailing and truncation, an adaptive attenuation window is added to the reconstructed tail segment. The termination condition is jointly triggered by an energy threshold and semantic boundaries, ultimately yielding the target speech output.

[0083] For example, in the two-stage framework of non-autoregressive models, pseudo-autoregressive ideas are introduced to enhance the time series modeling capability of the first stage, and direct preference optimization is introduced to align and improve the final output quality of the second stage.

[0084] Temporally Aware Semantic Token Prediction Based on Pseudo-Autoregression (Improved Text-to-Semantic Stage): This improvement modifies the training and inference methods of the first stage (text-to-semantic) of the non-autoregressive model. Instead of random masking, it adopts continuous segment masking as used in pseudo-autoregressive models. During training, the mask starts from a random position S and extends to the end of the sequence, but the model is only required to predict the leftmost segment (e.g., 10%) of the masked portion. During inference, a fully masked sequence is initialized, and the model predicts all positions in parallel at each step, but only retains and submits the leftmost K ungenerated tokens. The next step then predicts the next K tokens based on the submitted tokens, and so on. This improvement injects a strong temporal inductive bias into the fully parallel non-autoregressive model. The model is forced to learn "to predict the most likely semantic content to appear next given the preceding context," which is more in line with the nature of speech generation and can significantly improve the accuracy and robustness of content alignment while reducing word error rate.

[0085] A human-preference-based acoustic token optimizer (direct preference optimization fine-tuning the semantic-to-acoustic stage): First, a preference dataset is constructed. For the same text and cue speech, multiple outputs are generated using the original non-autoregressive model. These outputs are ranked by word error rate and similarity, and the best is selected as the preferred data, while the worst is selected as the unpreference data. The second stage of the non-autoregressive model (semantic-to-acoustic model) is considered as the policy model to be optimized. Its input is the semantic token and the cue acoustic token, and its output is the probability distribution of the acoustic token. Using the prepared preference data, the semantic-to-acoustic model is fine-tuned directly through the direct preference optimization loss function. The goal of direct preference optimization is to maximize the probability of "winning" samples relative to "losing" samples in the preference data without training a separate reward model. The loss function is as follows:

[0086] in, Represents the overall loss function (the loss value of Direct Preference Optimization), a scalar that is to be minimized during training; This represents the expectation of the distribution of the preference data. , , ) is a triplet sample obtained from the preference dataset: x: Input conditions, which are the input conditions for the "semantic-to-acoustic model", generally including text, prompt speech, etc. The output marked as "better" in this sample (winner, preferred sample); The output marked as "worse" in this sample (loser, unfavorable sample).

[0087] This represents the conditional probability distribution given by the current "policy model" (i.e., the semantic-to-acoustic model to be fine-tuned), and represents the probability of generating output y given input x. These are the trainable parameters of this model; This represents the conditional probability distribution of the reference model. It is usually the initial model before fine-tuning or a frozen baseline model. The parameters remain unchanged during training to provide a log probability baseline. , This indicates that the policy model, given input x, treats the preference sample... Non-preference samples The logarithmic probability; , This represents the log probability of the reference model for the preferred and unpreferred samples under the same input x; This represents a positive scaling hyperparameter used to control the degree to which the log probability difference is "amplified" in the loss; the larger the value, the "sharper" the preference signal. ( ) represents the Sigmoid function.

[0088] This represents the increase in log probability of the policy model on the preferred samples compared to the reference model; This represents the increase in log probability of the policy model relative to the reference model on unfavorable samples; these two differences are expressed as... After scaling and combining, the samples are fed into the Sigmoid function, which is equivalent to comparing "the degree to which the policy model favors preferred samples over non-preferred samples".

[0089] This embodiment achieves a stable mapping from semantics to timbre and prosody under the continuous constraint of prompting speech conditions through phased reasoning from text to semantics and from semantics to acoustics. The text-to-semantics model trained by pseudo-autoregression provides a temporally ordered sequence of semantic tokens, and the semantic-to-acoustics model optimized by preferences internalizes human ear preferences into an acoustic token distribution. The vocoder completes high-fidelity reconstruction. The three work together to reduce word error rate and pronunciation drift, improve speaker consistency and the naturalness of pauses and rhythms, and control tail trailing and splicing artifacts to obtain a clear, coherent and stylistically appropriate target speech output.

[0090] In one embodiment, a speech generation device based on pseudo-autoregressive modeling is provided, which corresponds one-to-one with the speech generation method based on pseudo-autoregressive modeling in the above embodiments. (Refer to...) Figure 3 , Figure 3This is a schematic diagram of the functional modules of a preferred embodiment of the speech generation device based on pseudo-autoregressive modeling of the present invention. The modules include a training sample construction module 10, a text-to-semantic training module 20, a candidate speech generation module 30, a preference data construction module 40, a semantic-to-acoustic optimization module 50, and a speech generation module 60. Detailed descriptions of each functional module are as follows: Training sample construction module 10 is used to obtain training samples containing text sequences, prompt speech fragments, and target semantic token sequences; The text-to-semantic training module 20 is used to train the text-to-semantic model based on the training samples. During training, a continuous segment masking strategy is applied to the target semantic token sequence, and the text-to-semantic model is trained to predict only the leftmost segment of the masked part, thus obtaining a text-to-semantic model trained by pseudo-autoregression. The candidate speech generation module 30 is used to process the training samples using the text-to-semantic model trained by pseudo-autoregression and the initial semantic-to-acoustic model to generate multiple candidate speech outputs. The preference data construction module 40 is used to sort the candidate speech outputs based on the word error rate and similarity index to construct preference data pairs; The semantic-to-acoustic optimization module 50 is used to update the parameters of the initial semantic-to-acoustic model based on the preference data pair using the direct preference optimization module, so as to obtain a preference-optimized semantic-to-acoustic model. The speech generation module 60 is used to acquire target text and target prompt speech, and process the target text and target prompt speech through the text-to-semantic model trained by pseudo-autoregression and the semantic-to-acoustic model optimized by preference to obtain target speech output.

[0091] In one embodiment, the training sample construction module 10 is specifically used for: Obtain the original text data, perform word segmentation on the original text data, and obtain the word-segmented text sequence; The original prompt voice is obtained, and the original prompt voice is preprocessed and segmented to obtain preprocessed prompt voice segments; The preprocessed prompt speech segment is processed using an acoustic feature extractor to obtain an acoustic feature sequence; The segmented text sequence and the acoustic feature sequence are input into a pre-trained semantic token generation model to obtain the target semantic token sequence; The segmented text sequence, the preprocessed prompt speech segment, and the target semantic token sequence are combined to obtain training samples.

[0092] In one embodiment, the text-to-semantic training module 20 is specifically used for: For the target semantic token sequence, a mask start position is randomly generated; Starting from the mask start position, the continuous segments in the target semantic token sequence up to the end of the sequence are masked to generate the masked semantic token sequence. The text sequence, the prompt speech segment, and the masked semantic token sequence are combined into model input data; The model input data is fed into the semantic model, and forward propagation is performed to obtain the prediction result of the mask position token. Extract the predicted value corresponding to the leftmost segment of the masked continuous segment from the prediction results; Determine the loss value between the predicted value of the leftmost segment and the real token in the target semantic token sequence; Based on the loss value, the parameters of the text-to-semantic model are updated through the backpropagation module until a preset training termination condition is reached, thus obtaining a text-to-semantic model trained by pseudo-autoregression.

[0093] In one embodiment, the candidate speech generation module 30 is specifically used for: The text sequence and the prompt voice segment are input into the pseudo-autoregressive trained text into the semantic model to generate a semantic token sequence; Set a random seed, input the semantic token sequence and the prompt speech segment into the initial semantic-to-acoustic model, and generate an acoustic token sequence based on the random seed; The acoustic token sequence is synthesized into candidate speech waveforms; The steps of setting a random seed, generating an acoustic token sequence, and synthesizing candidate speech waveforms are repeated multiple times to obtain multiple candidate speech waveforms. The multiple candidate speech waveforms are output as multiple candidate speech outputs.

[0094] In one embodiment, the preference data construction module 40 is specifically used for: Each candidate speech output is input into the automatic speech recognition module to obtain the speech-recognized text. The speech recognition text corresponding to each candidate speech output is compared with the corresponding text sequence to determine the word error rate value; Extract the acoustic feature vector of each candidate speech output, and extract the acoustic feature vector of the corresponding prompt speech segment for each candidate speech output; Determine the cosine similarity value between the acoustic feature vector of each candidate speech output and the acoustic feature vector of the corresponding prompt speech segment; The word error rate values ​​of all candidate speech outputs are normalized to obtain word error rate scores; The cosine similarity values ​​of all candidate speech outputs are normalized to obtain the similarity score; The word error rate score and similarity score of each candidate speech output are weighted and summed to obtain the comprehensive score; All candidate speech outputs are sorted from highest to lowest based on the comprehensive score; The candidate speech output with the highest overall score is selected as the preferred sample from the ranking results; The candidate speech output with the lowest overall score is selected as the unbiased sample from the ranking results; The preferred sample and the non-preferred sample are combined into a preference data pair.

[0095] In one embodiment, the semantic-to-acoustic optimization module 50 is specifically used for: The initial semantics are loaded into the acoustic model as a policy model, and the parameters of the policy model are copied to establish a reference model; Obtain training batches containing input conditions, preferred samples, and non-preferred samples from the preference data pairs; The input conditions and the preference samples are respectively input into the policy model and the reference model to obtain the first log probability of the policy model with respect to the preference samples and the third log probability of the reference model with respect to the preference samples. The input conditions and the unfavorable samples are respectively input into the policy model and the reference model to obtain the second log probability of the policy model for the unfavorable samples and the fourth log probability of the reference model for the unfavorable samples. Determine a first difference between the first logarithmic probability and the third logarithmic probability, and a second difference between the second logarithmic probability and the fourth logarithmic probability; The loss value is determined by using the direct preference optimization loss function based on the first difference and the second difference. Based on the loss value, the gradient is determined through the backpropagation module, and the parameters of the policy model are updated based on the gradient. Repeat the operations of acquiring training batches, determining log probabilities, determining differences, and updating parameters until the training stopping condition is met. The updated policy model is then used as the preference-optimized semantic-to-acoustic model.

[0096] In one embodiment, the speech generation module 60 is specifically used for: Obtain the target text and the target prompt speech, input the target text and the target prompt speech into the semantic model trained by pseudo-autoregression, and generate a semantic token sequence; The semantic token sequence and the target prompt speech are input into the preference-optimized semantic-to-acoustic model to generate an acoustic token sequence; The acoustic token sequence is input into a vocoder, which converts the acoustic token sequence into a speech waveform signal to obtain the target speech output.

[0097] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides deterministic and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side speech generation method based on pseudo-autoregressive modeling.

[0098] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements client-side functions or steps of a speech generation method based on pseudo-autoregressive modeling.

[0099] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain training samples containing text sequences, prompt speech fragments, and target semantic token sequences; The text-to-semantic model is trained based on the training samples. During training, a continuous segment masking strategy is applied to the target semantic token sequence, and the text-to-semantic model is trained to predict only the leftmost segment of the masked part, resulting in a text-to-semantic model trained by pseudo-autoregression. The training samples are processed using the pseudo-autoregressive text-to-semantic model and the initial semantic-to-acoustic model to generate multiple candidate speech outputs. Based on the word error rate and similarity index of the candidate speech output, the candidate speech outputs are sorted to construct preference data pairs; Based on the preference data pairs, the parameters of the initial semantic-to-acoustic model are updated using the direct preference optimization module to obtain a preference-optimized semantic-to-acoustic model. The target text and target prompt speech are acquired, and then processed by the text-to-semantic model trained by pseudo-autoregression and the semantic-to-acoustic model optimized by preference to obtain the target speech output.

[0100] In one embodiment, a non-volatile computer-readable storage medium is provided, which may be non-volatile or volatile, and stores a computer program thereon. When the computer program is executed by a processor, it performs the following steps: Obtain training samples containing text sequences, prompt speech fragments, and target semantic token sequences; The text-to-semantic model is trained based on the training samples. During training, a continuous segment masking strategy is applied to the target semantic token sequence, and the text-to-semantic model is trained to predict only the leftmost segment of the masked part, resulting in a text-to-semantic model trained by pseudo-autoregression. The training samples are processed using the pseudo-autoregressive text-to-semantic model and the initial semantic-to-acoustic model to generate multiple candidate speech outputs. Based on the word error rate and similarity index of the candidate speech output, the candidate speech outputs are sorted to construct preference data pairs; Based on the preference data pairs, the parameters of the initial semantic-to-acoustic model are updated using the direct preference optimization module to obtain a preference-optimized semantic-to-acoustic model. The target text and target prompt speech are acquired, and then processed by the text-to-semantic model trained by pseudo-autoregression and the semantic-to-acoustic model optimized by preference to obtain the target speech output.

[0101] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0102] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0103] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0104] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0105] The user personal information involved in this application embodiment is all authorized (knowing and consenting) by the relevant parties or fully authorized by all parties, and the executing entity can obtain it through various open, legal and compliant means. The collection, storage, use, processing, transmission, provision and disclosure of the information, data and signals involved all comply with the relevant laws and regulations of the relevant countries and regions, and do not violate public order and good morals.

Claims

1. A speech generation method based on pseudo-autoregressive modeling, characterized in that, Includes the following steps: Obtain training samples containing text sequences, prompt speech fragments, and target semantic token sequences; The text-to-semantic model is trained based on the training samples. During training, a continuous segment masking strategy is applied to the target semantic token sequence, and the text-to-semantic model is trained to predict only the leftmost segment of the masked part, resulting in a text-to-semantic model trained by pseudo-autoregression. The training samples are processed using the pseudo-autoregressive text-to-semantic model and the initial semantic-to-acoustic model to generate multiple candidate speech outputs. Based on the word error rate and similarity index of the candidate speech output, the candidate speech outputs are sorted to construct preference data pairs; Based on the preference data pairs, the parameters of the initial semantic-to-acoustic model are updated using the direct preference optimization module to obtain a preference-optimized semantic-to-acoustic model. The target text and target prompt speech are acquired, and then processed by the text-to-semantic model trained by pseudo-autoregression and the semantic-to-acoustic model optimized by preference to obtain the target speech output.

2. The speech generation method based on pseudo-autoregressive modeling as described in claim 1, characterized in that, Obtain training samples containing text sequences, prompt speech fragments, and target semantic token sequences, including: Obtain the original text data, perform word segmentation on the original text data, and obtain the word-segmented text sequence; The original prompt voice is obtained, and the original prompt voice is preprocessed and segmented to obtain preprocessed prompt voice segments; The preprocessed prompt speech segment is processed using an acoustic feature extractor to obtain an acoustic feature sequence; The segmented text sequence and the acoustic feature sequence are input into a pre-trained semantic token generation model to obtain the target semantic token sequence; The segmented text sequence, the preprocessed prompt speech segment, and the target semantic token sequence are combined to obtain training samples.

3. The speech generation method based on pseudo-autoregressive modeling as described in claim 1, characterized in that, The text-to-semantic model is trained based on the training samples. During training, a continuous segment masking strategy is applied to the target semantic token sequence, and the text-to-semantic model is trained to predict only the leftmost segment of the masked portion, resulting in a text-to-semantic model trained via pseudo-autoregression, including: For the target semantic token sequence, a mask start position is randomly generated; Starting from the mask start position, the continuous segments in the target semantic token sequence up to the end of the sequence are masked to generate the masked semantic token sequence. The text sequence, the prompt speech segment, and the masked semantic token sequence are combined into model input data. The model input data is fed into the semantic model, and forward propagation is performed to obtain the prediction result of the mask position token. Extract the predicted value corresponding to the leftmost segment of the masked continuous segment from the prediction results; Determine the loss value between the predicted value of the leftmost segment and the real token in the target semantic token sequence; Based on the loss value, the parameters of the text-to-semantic model are updated through the backpropagation module until a preset training termination condition is reached, thus obtaining a text-to-semantic model trained by pseudo-autoregression.

4. The speech generation method based on pseudo-autoregressive modeling as described in claim 1, characterized in that, The training samples are processed using the pseudo-autoregressive text-to-semantic model and the initial semantic-to-acoustic model to generate multiple candidate speech outputs, including: The text sequence and the prompt voice segment are input into the pseudo-autoregressive trained text into the semantic model to generate a semantic token sequence; Set a random seed, input the semantic token sequence and the prompt speech segment into the initial semantic-to-acoustic model, and generate an acoustic token sequence based on the random seed; The acoustic token sequence is synthesized into candidate speech waveforms; The steps of setting a random seed, generating an acoustic token sequence, and synthesizing candidate speech waveforms are repeated multiple times to obtain multiple candidate speech waveforms. The multiple candidate speech waveforms are output as multiple candidate speech outputs.

5. The speech generation method based on pseudo-autoregressive modeling as described in claim 1, characterized in that, Based on the word error rate and similarity index of the candidate speech output, a preference data pair is constructed, including: Each candidate speech output is input into the automatic speech recognition module to obtain the speech-recognized text. The speech recognition text corresponding to each candidate speech output is compared with the corresponding text sequence to determine the word error rate value; Extract the acoustic feature vector of each candidate speech output, and extract the acoustic feature vector of the corresponding prompt speech segment for each candidate speech output; Determine the cosine similarity value between the acoustic feature vector of each candidate speech output and the acoustic feature vector of the corresponding prompt speech segment; The word error rate values ​​of all candidate speech outputs are normalized to obtain word error rate scores; The cosine similarity values ​​of all candidate speech outputs are normalized to obtain the similarity score; The word error rate score and similarity score of each candidate speech output are weighted and summed to obtain the comprehensive score; All candidate speech outputs are sorted from highest to lowest based on the comprehensive score; The candidate speech output with the highest overall score is selected as the preferred sample from the ranking results; The candidate speech output with the lowest overall score is selected as the unbiased sample from the ranking results; The preferred sample and the non-preferred sample are combined into a preference data pair.

6. The speech generation method based on pseudo-autoregressive modeling as described in claim 1, characterized in that, Based on the preference data pairs, the parameters of the initial semantic-to-acoustic model are updated using the direct preference optimization module to obtain a preference-optimized semantic-to-acoustic model, including: The initial semantics are loaded into the acoustic model as a policy model, and the parameters of the policy model are copied to establish a reference model; Obtain training batches containing input conditions, preferred samples, and non-preferred samples from the preference data pairs; The input conditions and the preference samples are respectively input into the policy model and the reference model to obtain the first log probability of the policy model with respect to the preference samples and the third log probability of the reference model with respect to the preference samples. The input conditions and the unfavorable samples are respectively input into the policy model and the reference model to obtain the second log probability of the policy model for the unfavorable samples and the fourth log probability of the reference model for the unfavorable samples. Determine a first difference between the first logarithmic probability and the third logarithmic probability, and a second difference between the second logarithmic probability and the fourth logarithmic probability; The loss value is determined by using the direct preference optimization loss function based on the first difference and the second difference. Based on the loss value, the gradient is determined through the backpropagation module, and the parameters of the policy model are updated based on the gradient. Repeat the operations of acquiring training batches, determining log probabilities, determining differences, and updating parameters until the training stopping condition is met. The updated policy model is then used as the preference-optimized semantic-to-acoustic model.

7. The speech generation method based on pseudo-autoregressive modeling as described in claim 1, characterized in that, The target text and target prompt speech are acquired, and then processed using the pseudo-autoregressive text-to-semantic model and the preference-optimized semantic-to-acoustic model to obtain the target speech output, including: Obtain the target text and the target prompt speech, input the target text and the target prompt speech into the semantic model trained by pseudo-autoregression, and generate a semantic token sequence; The semantic token sequence and the target prompt speech are input into the preference-optimized semantic-to-acoustic model to generate an acoustic token sequence; The acoustic token sequence is input into a vocoder, which converts the acoustic token sequence into a speech waveform signal to obtain the target speech output.

8. A speech generation device based on pseudo-autoregressive modeling, characterized in that, The speech generation device based on pseudo-autoregressive modeling includes: The training sample construction module is used to obtain training samples containing text sequences, prompt speech fragments, and target semantic token sequences; The text-to-semantic training module is used to train the text-to-semantic model based on the training samples. During training, a continuous segment masking strategy is applied to the target semantic token sequence, and the text-to-semantic model is trained to predict only the leftmost segment of the masked part, thus obtaining a text-to-semantic model trained by pseudo-autoregression. The candidate speech generation module is used to process the training samples using the text-to-semantic model trained by pseudo-autoregression and the initial semantic-to-acoustic model to generate multiple candidate speech outputs. The preference data construction module is used to sort the candidate speech outputs based on the word error rate and similarity index, and construct preference data pairs. The semantic-to-acoustic optimization module is used to update the parameters of the initial semantic-to-acoustic model based on the preference data pair using the direct preference optimization module, so as to obtain a preference-optimized semantic-to-acoustic model. The speech generation module is used to acquire target text and target prompt speech, and process the target text and target prompt speech through the text-to-semantic model trained by pseudo-autoregression and the semantic-to-acoustic model optimized by preference to obtain the target speech output.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a speech generation program based on pseudo-autoregressive modeling stored in the memory and executable on the processor. When executed by the processor, the speech generation program based on pseudo-autoregressive modeling implements the steps of the speech generation method based on pseudo-autoregressive modeling as described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that, The storage medium stores a speech generation program based on pseudo-autoregressive modeling, which, when executed by a processor, implements the steps of the speech generation method based on pseudo-autoregressive modeling as described in any one of claims 1-7.

Images