Speech synthesis model training method and speech synthesis method

Abstract

The embodiment of the specification provides a speech synthesis model training method and a speech synthesis method, wherein the speech synthesis model training method comprises: obtaining sample speech; encoding a reference speech segment to obtain reference speech features, and encoding a target speech segment to obtain positive sample speech features; encoding a target speech text corresponding to the target speech segment to obtain target text features, and calling a reference speech synthesis model to generate negative sample speech features according to the target text features; and adjusting parameters of an initial speech synthesis model based on the reference speech features, the target text features, the positive sample speech features and the negative sample speech features to obtain a trained speech synthesis model, so that the model obtains optimization signals in two directions of approaching the positive sample and moving away from the negative sample at the same time when the parameters are updated, thereby strengthening the compliance of the model to the reference audio condition and improving the style consistency between the synthesized speech and the reference audio.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to speech synthesis model training methods and speech synthesis methods. Background Technology

[0002] In recent years, two-stage speech synthesis technology based on large language models and acoustic decoders has gradually become a research hotspot. In this type of technology, the large language model is responsible for mapping text into a discrete acoustic tag sequence, and then the acoustic decoder restores the sequence into an audio waveform.

[0003] In the training process of this type of technology, reconstruction loss is usually used to supervise the training of large language models. That is, real target speech sequences are used as supervision signals to guide the large language model to learn the mapping relationship from text and reference audio to acoustic tag sequences.

[0004] However, this type of training method lacks a constraint mechanism on whether the large language model effectively utilizes the reference audio. The large language model may only rely on textual information to generate acoustically labeled sequences, ignoring the stylistic features in the reference audio, resulting in inconsistencies between the style of the synthesized speech and the reference audio. Furthermore, existing training methods rely on a single real target sequence for supervision, limiting the large language model's ability to follow the reference audio. Therefore, improving the ability of speech synthesis models to follow reference audio has become an urgent technical problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a method for training a speech synthesis model. One or more embodiments of this specification also relate to a speech synthesis method, a method for constructing speech synthesis training samples, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, a method for training a speech synthesis model is provided, comprising: Acquire sample speech, wherein the sample speech includes a reference speech segment and a target speech segment; The reference speech segment is encoded to obtain reference speech features, and the target speech segment is encoded to obtain positive sample speech features; The target speech text corresponding to the target speech segment is encoded to obtain target text features, and a reference speech synthesis model is called to generate negative sample speech features based on the target text features; Based on the reference speech features, the target text features, the positive sample speech features, and the negative sample speech features, the parameters of the initial speech synthesis model are adjusted to obtain the trained speech synthesis model.

[0007] According to a second aspect of the embodiments of this specification, a speech synthesis method is provided, comprising: Obtain the target reference audio and the text to be synthesized; The target reference audio is encoded to obtain reference speech features, and the text to be synthesized is encoded to obtain text features. The speech synthesis model is invoked to generate target synthesized speech based on the reference speech features and the text features to be synthesized, wherein the speech synthesis model is trained based on the method described in the first aspect.

[0008] According to a third aspect of the embodiments of this specification, a method for constructing speech synthesis training samples is provided, comprising: Acquire sample speech, wherein the sample speech includes a reference speech segment and a target speech segment; The reference speech segment is encoded to obtain reference speech features, and the target speech segment is encoded to obtain positive sample speech features; The target speech text corresponding to the target speech segment is encoded to obtain target text features, and a reference speech synthesis model is called to generate negative sample speech features based on the target text features; Based on the reference speech features, the target text features, the positive sample speech features, and the negative sample speech features, speech synthesis training samples are constructed.

[0009] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the methods provided in the first, second, or third aspects described above.

[0010] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in the first, second, or third aspects described above.

[0011] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in the first, second, or third aspects described above.

[0012] This specification provides an embodiment of a speech synthesis model training method. By acquiring sample speech containing a reference speech segment and a target speech segment, the consistency between the two in terms of speaker identity and stylistic background is ensured. The reference speech segment and the target speech segment are encoded separately to obtain reference speech features and positive sample speech features, ensuring the model can establish an effective mapping from reference conditions to the target output. The target speech text is encoded to obtain target text features, and the reference speech synthesis model is invoked to generate negative sample speech features based on the target text features, providing a benchmark for positive and negative sample comparison. Based on the reference speech features, target text features, positive sample speech features, and negative sample speech features, the parameters of the initial speech synthesis model are adjusted, constructing a preference constraint mechanism for positive and negative sample comparison. This allows the model to simultaneously obtain optimization signals in both directions—moving closer to positive samples and moving away from negative samples—during parameter updates, thereby strengthening the model's ability to follow reference audio conditions, overcoming the limitations of supervision by a single real target sequence, and improving the stylistic consistency between synthesized speech and reference audio. Attached Figure Description

[0013] Figure 1 This is a flowchart illustrating a speech synthesis model training method provided in one embodiment of this specification; Figure 2 This is a schematic diagram illustrating the process of a speech synthesis model training method provided in one embodiment of this specification; Figure 3 This is a flowchart illustrating a method for constructing training samples for speech synthesis, as provided in one embodiment of this specification. Figure 4 This is a flowchart illustrating a speech synthesis method provided in one embodiment of this specification; Figure 5 This is an architecture diagram of a speech synthesis system provided in one embodiment of this specification; Figure 6a This is a schematic diagram of the structure of a speech synthesis model training device provided in one embodiment of this specification; Figure 6b This is a schematic diagram of another speech synthesis model training device provided in one embodiment of this specification; Figure 7 This is a schematic diagram of the structure of a speech synthesis device provided in one embodiment of this specification; Figure 8 This is a schematic diagram of a speech synthesis training sample construction device provided in one embodiment of this specification; Figure 9 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0014] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0015] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items. The term “at least one” as used in one or more embodiments of this specification means “one or more,” and “a plurality of” means “two or more.” The term “comprising” is an open-ended description and should be understood as “including but not limiting,” and may include other content in addition to what has been described.

[0016] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0017] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in one or more embodiments of this specification are obtained through open-source datasets or public datasets that comply with their license agreements, or are obtained with full authorization from the relevant parties. Moreover, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0018] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model, which is pre-trained on a large-scale unlabeled corpus to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability, such as large-scale language models (LLMs) and multi-modal pre-training models.

[0019] In practical applications, large-scale models require only a small number of samples to fine-tune the pre-trained model before being applied to different tasks. Large-scale models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. Major application scenarios for large-scale models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0020] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0021] Acoustic tokens are discretized representations of speech signals. They are generated by a speech segmenter that encodes continuous audio signals into discrete sequences, carrying acoustic information such as prosody, style, and timbre. They serve as an intermediate representation between large language models and acoustic decoders, and are a key bridge connecting language model prediction and audio waveform generation.

[0022] Large Language Model: A large-scale pre-trained language model based on the Transformer architecture. It is an autoregressive language model used to map text into discrete acoustic token sequences. It gains language understanding and generation capabilities through pre-training on massive amounts of text data and can predict and generate target sequences token by token in an autoregressive manner.

[0023] Conditional Flow Matching (CFM) is an acoustic decoder technique that combines discrete acoustic token sequences with timbre vectors to reconstruct continuous audio waveforms. By modeling the conditional probability flow path from noise to target audio, it achieves high-quality speech waveform reconstruction and is the core module of the acoustic decoding stage in a two-stage speech synthesis architecture.

[0024] The speech tokenizer is a module that encodes continuous audio waveforms into discrete token sequences. By extracting semantic and acoustic features from the audio and quantizing them into discrete codebook indices, it achieves the discrete representation of audio signals. It is the core encoding and decoding component that connects the continuous audio space and the discrete token space in a speech synthesis system.

[0025] One-shot synthesis is a technique that uses only a reference audio clip to mimic the style of the speaker, without requiring additional model fine-tuning or training for the target speaker. It achieves zero-shot or few-shot style transfer across speakers by extracting features such as timbre and prosody from the reference audio and injecting them into the synthesis process.

[0026] Reference audio (Prompt Speech) and reference audio token (Prompt Speech Token): The reference audio refers to a short reference speech provided by the user, and the reference audio token refers to the token sequence after the speech is encoded by the speech tokenizer. It is used to guide the style and rhythm of the synthesized speech, so that the synthesized speech can imitate the timbre, speech rate, emotion and other features of the reference audio, and achieve one-shot style speech synthesis.

[0027] Direct Preference Optimization (DPO) is a training method for a policy model that directly optimizes the pair of positive and negative preferences. It constructs a loss function by maximizing the difference between the log-probability ratios of the policy model and the reference model on positive and negative examples. This achieves preference alignment without requiring a separate reward model, merging the reward modeling and policy optimization steps in reinforcement learning into a single closed-loop optimization objective.

[0028] The sigmoid function, a monotonically differentiable S-shaped function, maps any real input to the interval (0,1). It is smooth, continuous, and monotonically increasing, with its central symmetric point at (0,0.5). The sigmoid function converts the difference between positive and negative example preferences into probability values ​​within the (0,1) interval. A value closer to 1 indicates a larger preference gap and a stronger preference for positive examples over negative examples; a value closer to 0 indicates a smaller preference gap, or even a negative one. This transforms the comparison signal into a stable scalar loss, providing a smooth optimization surface for gradient backpropagation.

[0029] Text-to-speech (TTS) is a technology that converts text content into speech audio. The input is a sequence of text to be synthesized, and the output is the corresponding speech waveform. It is the core task in the field of speech synthesis and is widely used in scenarios such as smart assistants, audiobooks, and navigation broadcasts.

[0030] In a two-stage speech synthesis system based on a large language model and an acoustic decoder, the core modules typically include a large language model, a speech encoder, and an acoustic decoder. The large language model is a pre-trained autoregressive language model responsible for mapping text into discrete sequences of speech features. The speech encoder encodes continuous audio waveforms into discrete sequences of speech features. The acoustic decoder combines these speech feature sequences with timbre information to reconstruct the audio waveform.

[0031] In the speech synthesis process, the user provides a reference audio clip, expecting the synthesized speech to mimic the reference audio in terms of timbre, rhythm, and speaking style. The reference audio is encoded into reference speech features by a speech encoder, which are then concatenated with the encoded text features of the text to be synthesized to form conditional input features, which are then input into a large language model. The large language model progressively generates a sequence of speech features in an autoregressive manner, and finally, an acoustic decoder combines these features with the timbre vectors extracted from the reference audio to generate the synthesized speech.

[0032] Existing training methods typically use standard reconstruction losses to train large language models, such as cross-entropy or flow matching loss. During training, sample speech is acquired and split into reference and target speech segments. These segments are then encoded using a speech encoder to obtain reference speech features and positive sample speech features, respectively. The reference speech features are concatenated with the target text features corresponding to the target speech segment, and this concatenation is used as conditional input to the large language model. The large language model generates a predicted speech feature sequence using an autoregressive approach. The reconstruction loss between the predicted speech feature sequence and the positive sample speech features is then calculated, and the parameters of the large language model are updated via gradient backpropagation.

[0033] However, existing training methods rely solely on a single reconstruction loss for supervision, lacking explicit constraints on whether large language models truly utilize the reference speech conditional input. During training, large language models may generate speech feature sequences based solely on text features, ignoring prosody and speech style information in the reference speech features. This leads to spontaneous prosodic style in the synthesized speech, inconsistent with the style of the reference audio. Furthermore, existing methods rely only on a single real target sequence for supervision, lacking an effective monitoring mechanism for the extent to which the large language model adheres to the reference speech conditional input during training. Therefore, it is impossible to directly determine whether the reference speech conditional input is effectively utilized from the level of the model's output speech feature sequence.

[0034] To address the aforementioned problems, this specification provides a speech synthesis model training method. This specification also relates to a speech synthesis method, a speech synthesis training sample construction method, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0035] See Figure 1 , Figure 1 This specification illustrates a flowchart of a speech synthesis model training method according to an embodiment, which may specifically include the following steps: Step 102: Obtain sample speech, which includes a reference speech segment and a target speech segment.

[0036] It should be noted that sample speech refers to the speech data used to train the speech synthesis model, and is the basic data unit in the model training process. Sample speech can include various types, such as continuous speech recordings of real speakers, synthesized speech, and speech that has undergone data augmentation. Sample speech can be historical speech data pre-collected and filtered from the training corpus, or it can be training data obtained after preprocessing the original collected speech. For example, a 20-second continuous speech recording of a real single speaker can be selected from the training corpus as a sample speech.

[0037] A reference speech segment refers to a segment of speech in a sample speech that provides reference information on the speaker's style and prosody. The reference speech segment can be the first part of the sample speech taken chronologically, a specific segment in the middle of the sample speech, or a specific segment extracted based on semantic segmentation of the speech content. Its duration must meet the condition of containing complete speaking style information to ensure that subsequent models can extract sufficient style reference conditions from it. For example, in a 20-second continuous speech sample, the first 5 seconds of the speech segment can be used as the reference speech segment. This 5-second segment contains the speaker's complete timbre features, prosodic patterns, and speaking style.

[0038] The target speech segment refers to the speech fragment in the sample speech used to provide the model with the speech to learn and generate. It is a continuous speech segment belonging to the same speaker as the reference speech segment. The target speech segment can be the latter part of the sample speech segment excluding the reference speech segment, or it can be other segments in the sample speech segment that do not overlap with the reference speech segment, or it can be a segment corresponding to specific semantic content in the sample speech segment. The target speech segment and the reference speech segment are naturally coherent and have the same speaker identity and stylistic background. For example, in a 20-second continuous speech sample, the speech segment from the 5th second to the 20th second is taken as the target speech segment. This 15-second speech segment and the reference speech segment are the content spoken by the same speaker in this speech segment.

[0039] In practical applications, there are various ways to acquire sample speech, and the specific method should be selected according to the actual situation. This specification does not impose any limitations on this method. One possible implementation involves selecting continuous speech samples that meet a preset duration condition from the training corpus and dividing them into a first segment and a second segment according to chronological order, serving as the reference segment and the target segment, respectively. Another possible implementation involves acquiring two independent speech segments, one as the reference segment and the other as the target segment. These two segments can come from different sources, as long as the reference segment contains complete speaking style information.

[0040] For example, assume the training corpus contains a large number of real-world recordings of single-speaker speech. Select 20-second continuous speech samples from the training corpus and segment them chronologically into a 5-second reference segment and a 15-second target segment. The reference segment is set to approximately 5 seconds in length. This ensures it contains complete timbre features, prosodic patterns, and speaking style information, providing sufficient style reference for the model. Furthermore, by prioritizing segmentation at semantic boundaries, word truncation is avoided, guaranteeing the speech integrity of both the reference and target segments. Since the reference and target segments originate from the same speaker's continuous speech, they naturally possess stylistic and prosodic consistency.

[0041] Step 104: Encode the reference speech segment to obtain reference speech features, and encode the target speech segment to obtain positive sample speech features.

[0042] It should be noted that reference speech features refer to the discrete feature representation obtained after encoding a reference speech segment. They are the digital representation of the reference speech segment in the representation space that the model can process, carrying information such as timbre, prosody, and speaking style. Reference speech features can be a sequence of discrete speech markers obtained by encoding continuous reference speech segments using a speech encoder. Reference speech features are used to provide style reference conditions for speech synthesis models, enabling the model to mimic the timbre and prosody of the reference speech to generate speech. For example, a 5-second reference speech segment can be encoded into a sequence of 200 discrete speech markers by a speech encoder, where each marker corresponds to the acoustic features of a specific moment in the reference speech segment.

[0043] Positive sample speech features refer to the discrete feature representation obtained after encoding a target speech segment. They are the digital representation of the target speech segment in the model's processable representation space, representing the target speech that the model needs to learn to generate during training. Positive sample speech features can be a sequence of discrete speech tokens obtained by encoding continuous target speech segments using a speech encoder. Positive sample speech features are used as the generation target that the model needs to approach during training. For example, a 15-second target speech segment can be encoded into a sequence of 600 discrete speech tokens using a speech encoder. Each token in this sequence corresponds to the true acoustic features of the target speech segment at a specific moment.

[0044] In practical applications, there are various ways to encode a reference speech segment to obtain reference speech features. The specific method chosen depends on the actual situation, and the embodiments in this specification do not impose any limitations on this. One possible implementation in this specification is to call the same speech encoder to encode both the reference speech segment and the target speech segment separately, obtaining reference speech features and positive sample speech features, ensuring that both are in the same representation space. Another possible implementation in this specification is to use the feature extraction layer of an acoustic model to encode the reference speech segment, extracting its deep acoustic representation, and then converting it into a discrete feature sequence through a linear mapping layer, which serves as the reference speech feature.

[0045] In practical applications, there are various ways to encode a target speech segment to obtain positive sample speech features. The specific method chosen depends on the actual situation, and this specification does not limit the implementation in any way. One possible implementation involves using the same speech encoder as the reference speech segment to encode the target speech segment and obtain positive sample speech features, ensuring that the positive sample speech features and the reference speech features are in the same representation space. Another possible implementation involves using a completely different encoding model than the reference speech segment to encode the target speech segment. For example, a pre-trained accent recognition model or language recognition model's feature extraction layer can be used to encode the target speech segment, extracting its acoustic representation as positive sample speech features, and then mapping it to the same representation space as the reference speech features through a feature alignment layer.

[0046] For example, suppose the sample speech is a 20-second recording of a real single-speaker continuous speech, segmented according to semantic boundaries into a 5-second reference speech segment and a 15-second target speech segment. The same speech encoder is used to encode both the reference and target speech segments separately. After encoding, the 5-second continuous audio signal of the reference speech segment is converted into a sequence of 200 discrete speech markers, serving as the reference speech features. After encoding the target speech segment using the same speech encoder, the 15-second continuous audio signal is converted into a sequence of 600 discrete speech markers, serving as the positive sample speech features. Both are obtained through the same speech encoder and reside in the same representation space.

[0047] In one optional embodiment of this specification, the above-described encoding of the reference speech segment to obtain reference speech features and encoding of the target speech segment to obtain positive sample speech features may include the following steps: The same speech encoder is used to encode the reference speech segment and the target speech segment respectively, so as to obtain the reference speech features and the positive sample speech features.

[0048] It's important to note that a speech encoder refers to a module that uses the same SpeechTokenizer to encode both the reference and target speech segments. This same speech encoder can be a discrete speech coding model trained on a neural network, encoding continuous audio waveforms into discrete acoustic token sequences. Using the same speech encoder ensures that the features output from different speech segments lie in the same representation space, allowing subsequent models to directly calculate and compare the probabilities of the reference speech features and the features of positive sample speech. For example, both the reference and target speech segments can be encoded into discrete acoustic token sequences using the same SpeechTokenizer, where each token represents the acoustic feature at that specific moment.

[0049] In practical applications, there are multiple ways to implement the same speech encoder to encode the reference speech segment and the target speech segment separately. The specific method should be selected according to the actual situation, and the embodiments in this specification do not limit this approach. In one possible implementation, the reference speech segment and the target speech segment can be input sequentially into the same SpeechTokenizer to obtain the reference speech feature sequence and the positive sample speech feature sequence, respectively. In another possible implementation, the reference speech segment and the target speech segment can be grouped into a batch and simultaneously input into the same SpeechTokenizer for encoding in a batch processing manner. Batch inference is used to improve encoding efficiency, and the corresponding feature sequences are extracted separately after encoding.

[0050] For example, suppose the sample speech is a 20-second recording of a real single-speaker continuous speech, with the first 5 seconds as the reference speech segment and the last 15 seconds as the target speech segment. The first 5 seconds of the reference speech segment are input into a Speech Tokenizer, encoding a reference speech feature sequence consisting of 200 discrete speech tokens; the last 15 seconds of the target speech segment are input into the same Speech Tokenizer, encoding a positive sample speech feature sequence consisting of 600 discrete speech tokens. Since the same speech encoder is used for both encodings, the two sets of feature sequences reside in the same discrete feature space.

[0051] By applying the scheme of the embodiments in this specification, the reference speech segment and the target speech segment are encoded separately by calling the same speech encoder, which ensures that the reference speech features and the positive sample speech features are in a unified representation space, avoids the inconsistency problem of feature space introduced by different encoders, and enables the subsequent preference optimization loss calculation and model parameter adjustment to be performed accurately and effectively.

[0052] Step 106: Encode the target speech text corresponding to the target speech segment to obtain the target text features, and call the reference speech synthesis model to generate negative sample speech features based on the target text features.

[0053] It should be noted that target speech text refers to the textual content corresponding to a target speech segment; it is the text transcription of the speech content within the target speech segment. Target speech text can be manually annotated transcribed text, text recognized by an automatic speech recognition system, or text obtained in real-time through a dictation tool. Target speech text is used to convert speech content into a textual form that can be processed by a text encoder. For example, if a speaker in a target speech segment delivers a weather forecast, the corresponding textual content obtained through manual transcription or automatic speech recognition serves as the target speech text.

[0054] Target text features refer to the discrete feature representation obtained after encoding the target speech text; they are the digital expression of the target speech text in the representation space that the model can process. Target text features can be a sequence of discrete text tags obtained by encoding the text content using a text encoder, a sequence of semantic feature vectors output by the embedding layer of a pre-trained language model, or a sequence of phoneme tags obtained based on pinyin or phoneme annotation. For example, a text encoder can encode the target speech text into a sequence of 500 discrete text tags, where each tag corresponds to a word or phoneme at a specific position in the text.

[0055] A reference speech synthesis model refers to a pre-trained speech synthesis model whose parameters remain frozen during preference optimization training. The reference speech synthesis model can be a copy of the model weights before training begins, a model whose parameters are frozen after supervised fine-tuning, or a pre-trained speech synthesis model imported from an external source. For example, a reference speech synthesis model can be a large language model weight copy frozen before preference optimization training begins, used only for forward propagation calculations during training, and not subject to gradient updates.

[0056] Negative sample speech features refer to speech features generated by a reference speech synthesis model based on target text features without concatenating reference speech features. They represent the model's spontaneous output in the absence of style reference. Negative sample speech features can be discrete speech tag sequences. They are used as negative examples to be suppressed during preference optimization training, forming a contrast signal with positive sample speech features. For example, a discrete speech tag sequence generated by the reference speech synthesis model in an autoregressive manner based solely on target text features without concatenating reference speech features.

[0057] In practical applications, there are various ways to encode the target speech text corresponding to the target speech segment to obtain target text features. The specific method chosen depends on the actual situation, and the embodiments in this specification do not impose any limitations on this. In one possible implementation, a text encoder can be used to encode the target speech text into a discrete text tag sequence as the target text feature. In another possible implementation, the target speech text can first be converted into a pinyin sequence or phoneme sequence, and then the phoneme sequence can be encoded into a discrete tag sequence using a phoneme encoder as the target text feature. This method can eliminate the influence of differences in Chinese character shapes on the encoding result.

[0058] In practical applications, there are various ways to call a reference speech synthesis model to generate negative sample speech features based on the target text features. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. One possible implementation involves inputting the target text features into the reference speech synthesis model without concatenating the reference speech features, allowing the model to generate negative sample speech features in an autoregressive manner. Another possible implementation involves applying different noise perturbations or temperature parameters to the target text features without concatenating the reference speech features, and then repeatedly calling the reference speech synthesis model to generate negative sample speech features. This allows for the acquisition of diverse negative samples by controlling the differences in generation conditions.

[0059] For example, suppose the target speech segment is a 15-second continuous speech, and the corresponding transcribed target speech text describes a travel experience. The target speech text is encoded into a sequence of 500 discrete text tags using a text encoder, serving as the target text feature. Subsequently, without concatenating with the reference speech features, the target text feature is input into the reference speech synthesis model, which was frozen before the start of preference optimization training. The reference speech synthesis model then generates a sequence of 600 discrete speech tags in an autoregressive manner, serving as the negative sample speech feature. Since no style reference is provided during the generation of the negative sample speech feature, it reflects the spontaneous prosodic style of the model, relying solely on the semantic information of the text.

[0060] Step 108: Based on the reference speech features, target text features, positive sample speech features, and negative sample speech features, adjust the parameters of the initial speech synthesis model to obtain the trained speech synthesis model.

[0061] It's important to note that the initial speech synthesis model refers to the speech synthesis model that needs to be trained, and its parameters are updated during training. The initial speech synthesis model can be a large language model that has undergone basic pre-training, serving as the current training policy during the preference optimization phase. Iterative updates of parameters using preference optimization loss improve the model's ability to adhere to reference speech conditions. Alternatively, the initial speech synthesis model can be an intermediate version model that has undergone supervised fine-tuning, further improved for style consistency through preference optimization. The initial speech synthesis model receives conditional input features and generates speech feature sequences in an autoregressive manner. For example, the initial speech synthesis model might be a large language model that has undergone basic pre-training, responsible for receiving conditional input features and generating speech feature sequences token-by-token in an autoregressive manner.

[0062] A fully trained speech synthesis model refers to a speech synthesis model whose parameters have converged after preference optimization training. Compared to the initial speech synthesis model, it has a stronger ability to follow the style of the reference speech. A fully trained speech synthesis model can be used for practical speech synthesis inference tasks, or it can serve as the base model for subsequent training rounds for further optimization. For example, after multiple rounds of training, the parameters of the initial speech synthesis model are updated, resulting in a fully trained speech synthesis model capable of generating consistent speech based on the style of the reference audio.

[0063] In practical applications, there are various ways to adjust the parameters of the initial speech synthesis model based on reference speech features, target text features, positive sample speech features, and negative sample speech features to obtain the trained speech synthesis model. The specific implementation depends on the actual situation, and this specification does not limit this approach. One possible implementation involves concatenating reference speech features and target text features to generate conditional input features. The probability information of positive and negative sample speech features under the initial and reference speech synthesis models is then calculated, and a preference optimization loss is constructed to update the parameters of the initial speech synthesis model. Another possible implementation involves supervised training of the initial speech synthesis model based on the standard reconstruction loss, while simultaneously introducing a contrast constraint between positive and negative samples for joint optimization. This involves adding a penalty term for the probability difference between positive and negative samples to the negative log-likelihood loss.

[0064] For example, suppose we are currently in the training phase of a speech synthesis model. Reference speech features (a sequence of 200 discrete speech tokens) and target text features (a sequence of text tokens) are input as conditional inputs to the initial speech synthesis model. Based on this conditional input, the initial speech synthesis model generates a sequence of predicted speech features. The parameters of the initial speech synthesis model are then adjusted using positive sample speech features (the target that the model should generate) and negative sample speech features (the output that the model should avoid), allowing the model to gradually learn to generate speech features consistent with the reference style given the reference speech conditions. After multiple rounds of iterative training, the parameters of the initial speech synthesis model converge, resulting in a fully trained speech synthesis model.

[0065] In one optional embodiment of this specification, the above-described parameter adjustment of the initial speech synthesis model based on reference speech features, target text features, positive sample speech features, and negative sample speech features to obtain a trained speech synthesis model may include the following steps: Based on the reference speech features and the target text features, conditional input features are generated; Based on the conditional input features, the first positive probability information of positive sample speech features under the initial speech synthesis model and the first negative probability information of negative sample speech features under the initial speech synthesis model are determined. Based on conditional input features, the second positive probability information of positive sample speech features under the reference speech synthesis model and the second negative probability information of negative sample speech features under the reference speech synthesis model are determined. Based on the first positive probability information, the first negative probability information, the second positive probability information, and the second negative probability information, determine the preference optimization loss; Based on the preference optimization loss, the parameters of the initial speech synthesis model are adjusted to obtain the trained speech synthesis model.

[0066] It should be noted that conditional input features refer to the feature sequence formed by combining reference speech features and target text features in a preset manner, which serves as the complete conditional input for the initial speech synthesis model to generate speech features. Conditional input features can be a sequence of markers formed by concatenating reference speech features before target text features, a mixed sequence formed by alternating reference speech features and target text features, or a fused feature obtained by mapping both to the same dimension through a projection layer and then concatenating or adding the vectors. Conditional input features are used to simultaneously provide the model with style reference information and text semantic information, enabling the model to generate speech features that meet the requirements based on both. For example, a reference speech feature composed of 200 discrete speech markers can be concatenated before a text marker sequence to form a conditional input feature.

[0067] The first positive probability information refers to the probability assessment result of the model on positive sample speech features after the conditional input features are input into the initial speech synthesis model. The first positive probability information can be the product of the conditional probabilities of each sub-feature in the positive sample speech feature sequence by the initial speech synthesis model, or it can be the sum of the logarithms of the conditional probabilities at each position, or it can be the probability value after temperature scaling or other normalization processing. The first positive probability information is used to reflect the probability that the model in the current training stage will generate realistic target speech under reference conditions. For example, the initial speech synthesis model progressively predicts the features at each position in the positive sample speech feature sequence in an autoregressive manner, and the first positive probability information is obtained by summing the logarithms of the conditional probabilities at each position.

[0068] The first negative probability information refers to the probability assessment result of the initial speech synthesis model on the negative sample speech features after the conditional input features are input into the initial speech synthesis model. The first negative probability information can be the product of the conditional probabilities of each sub-feature in the negative sample speech feature sequence by the initial speech synthesis model, the sum of the logarithms of the conditional probabilities at each position, or the probability value after temperature scaling or other normalization processing. The first negative probability information is used to reflect the probability that the model currently being trained will generate spontaneous speech under reference conditions. For example, the initial speech synthesis model progressively predicts the features at each position in the negative sample speech feature sequence in an autoregressive manner, and the first negative probability information is obtained by summing the logarithms of the conditional probabilities at each position.

[0069] The second positive probability information refers to the probability assessment result of the reference speech synthesis model on positive sample speech features after the conditional input features are input into the reference model. The second positive probability information can be the product of the conditional probabilities of each sub-feature in the positive sample speech feature sequence by the reference speech synthesis model, or it can be the sum of the logarithms of the conditional probabilities at each position. The second positive probability information is used as a benchmark to measure the probability change of the current trained model relative to the reference model on positive samples. For example, the reference speech synthesis model progressively predicts the feature at each position in the positive sample speech feature sequence in an autoregressive manner, and the second positive probability information is obtained by summing the logarithms of the conditional probabilities at each position.

[0070] The second negative probability information refers to the probability assessment result of the reference speech synthesis model on negative sample speech features after the conditional input features are input into the reference model. The second negative probability information can be the product of the conditional probabilities of each sub-feature in the negative sample speech feature sequence by the reference speech synthesis model, or it can be the sum of the logarithms of the conditional probabilities at each position. The second negative probability information is used as a benchmark to measure the probability change of the current trained model relative to the reference model on negative samples. For example, the reference speech synthesis model progressively predicts the feature at each position in the negative sample speech feature sequence in an autoregressive manner, and the second negative probability information is obtained by summing the logarithms of the conditional probabilities at each position.

[0071] Preference optimization loss is a loss function value calculated based on the probability difference between positive and negative samples in the current training model and the reference model. It guides the parameter update direction of the initial speech synthesis model. Preference optimization loss can be the binary cross-entropy loss calculated based on the DPO algorithm, or the loss calculated based on the Identity Preference Optimization (IPO) algorithm. Preference optimization loss aims to increase the model's probability advantage in generating positive samples relative to negative samples, while using the probability of the reference model as a regularization constraint to prevent excessive deviation. For example, the preference optimization loss can be obtained by calculating the difference between the logarithmic probability ratios of the policy model and the reference model on positive and negative examples, then applying a sigmoid function mapping to calculate the binary cross-entropy.

[0072] In practical applications, there are multiple ways to determine the preference optimization loss based on the first positive probability information, the first negative probability information, the second positive probability information, and the second negative probability information. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. One possible implementation in this specification involves calculating the difference between the log-probability ratio of the policy model on positive and negative examples and the log-probability ratio of the reference model on positive and negative examples, and then calculating the preference optimization loss using the sigmoid function and binary cross-entropy. Another possible implementation in this specification involves adding a KL divergence constraint as a regularization term to the preference optimization loss. By constraining the distribution difference between the policy model and the reference model, the stability of model parameter updates during training is further enhanced, preventing the model from deviating excessively from its basic synthetic capabilities.

[0073] For example, suppose we are currently in one iteration of preference optimization training. The reference speech features (a sequence of 200 discrete speech tags) and the target text features (a sequence of text tags) are concatenated into conditional input features, which are then input into the initial speech synthesis model and the reference speech synthesis model with its parameters frozen, respectively. The initial speech synthesis model calculates a first positive probability for positive sample speech features (600 discrete speech tags) and a first negative probability for negative sample speech features (600 discrete speech tags); the reference speech synthesis model calculates a second positive probability for the same positive sample speech feature and a second negative probability for the same negative sample speech feature.

[0074] The scheme implemented in this specification, by concatenating reference speech features and target text features into conditional input features, enables the model to simultaneously utilize style reference information and text semantic information during the generation process, thereby generating speech features that are consistent with the style of the reference audio and have correct content. By calculating the first positive probability information and the first negative probability information of positive and negative samples under the current training model, and the second positive probability information and the second negative probability information under the reference model, precise quantification of the model's preference direction is achieved. Parameter updates are performed by constructing a preference optimization loss based on the four probabilistic information, achieving preference learning without separately training a reward model, simplifying the training process. Simultaneously, the probability of the reference model serves as a regularization benchmark, constraining the magnitude of parameter updates during training, preventing the trained model from deviating excessively from its original basic capabilities, and ensuring that the model maintains basic speech synthesis quality while improving its style adherence ability.

[0075] In one optional embodiment of this specification, determining the preference optimization loss based on the first positive probability information, the first negative probability information, the second positive probability information, and the second negative probability information may include the following steps: Compare the first positive probability information and the first negative probability information to determine the first probability difference; By comparing the second positive probability information with the second negative probability information, the second probability difference is determined. The preference optimization loss is determined based on the first probability difference and the second probability difference.

[0076] It should be noted that the first probability difference refers to a measure of the difference between the first positive probability information and the first negative probability information, representing the probabilistic advantage of the initial speech synthesis model in the current training process in generating positive samples relative to negative samples under reference conditions. The first probability difference can be the difference between the logarithmic probabilities of the first positive and first negative probability information, or it can be the difference calculated after normalizing both, or it can be the value obtained by logarithmically transforming their ratio. The first probability difference is used to quantify the discriminative ability of the current model on preference pairs. For example, the first positive probability information is the cumulative logarithmic probability of the initial speech synthesis model for positive sample speech features, and the first negative probability information is the cumulative logarithmic probability of the initial speech synthesis model for negative sample speech features; subtracting the two yields the first probability difference.

[0077] The second probability difference is a measure of the difference between the second positive probability information and the second negative probability information, representing the probabilistic advantage of the reference speech synthesis model in generating positive samples relative to negative samples under reference conditions. The second probability difference can be the difference between the logarithmic probabilities of the second positive and second negative probability information, the difference calculated after normalization of both, or the value obtained by logarithmically transforming their ratio. The second probability difference is used to provide a baseline reference before training. For example, the second positive probability information is the accumulated logarithmic probability of the reference speech synthesis model for positive sample speech features, and the second negative probability information is the accumulated logarithmic probability of the reference speech synthesis model for negative sample speech features; subtracting the two yields the second probability difference.

[0078] In practical applications, there are multiple ways to determine the preference optimization loss based on the first probability difference and the second probability difference. The specific method chosen depends on the actual situation, and this specification does not impose any limitations on this approach. In one possible implementation, the difference between the first and second probability differences can be calculated, scaled by a temperature coefficient, and then the preference optimization loss can be calculated using the sigmoid function and binary cross-entropy. In another possible implementation, the relative entropy between the policy model and the reference model can be used as a regularization term based on the first and second probability differences to constrain the offset of the policy distribution during model parameter updates, thereby enhancing training stability while ensuring the effectiveness of preference learning.

[0079] For example, suppose that in a certain iteration of preference optimization training, the initial speech synthesis model calculates the log probabilities of positive and negative sample speech features respectively, and subtracts the two probabilities to obtain the first probability difference, representing the current model's probabilistic advantage over negative samples under the condition of having a reference. Simultaneously, the frozen reference speech synthesis model calculates the log probabilities of the same positive and negative samples respectively, and subtracts the two probabilities to obtain the second probability difference, representing the probabilistic advantage of the reference model before training under the same conditions. The difference between the first and second probability differences is taken, scaled by a temperature coefficient, and then used to calculate the preference optimization loss through the sigmoid function.

[0080] The scheme implemented in this specification determines a first probability difference by comparing first positive probability information and first negative probability information, transforming the model's absolute probability assessment of positive and negative samples into a relative difference metric, thus eliminating the influence of the absolute probability scale on the optimization objective. A second probability difference is determined by comparing second positive probability information and second negative probability information, obtaining another set of relative difference metrics in the same way, ensuring comparable consistency between the two sets of data. Furthermore, the preference optimization loss is determined based on the first and second probability differences, and a loss function is constructed by combining the two probability differences. This allows for the determination of the optimization direction from multi-dimensional probability differences, enabling the preference optimization process to more accurately reflect the model's ability to distinguish between positive and negative samples.

[0081] In one optional embodiment of this specification, the number of negative sample speech features is multiple. Determining the preference optimization loss based on the first positive probability information, the first negative probability information, the second positive probability information, and the second negative probability information may include the following steps: Based on the first positive probability information, the first negative probability information, the second positive probability information, and the second negative probability information corresponding to the speech features of each negative sample, multiple candidate preference optimization losses are determined respectively, wherein the candidate preference optimization loss corresponds one-to-one with the speech features of the negative sample. The mean of the optimization losses of multiple candidate preferences is calculated to obtain the preference optimization loss.

[0082] It should be noted that the candidate preference optimization loss refers to the preference optimization loss value calculated for a single negative sample speech feature, combining the corresponding first positive probability information, first negative probability information, second positive probability information, and second negative probability information. There is a one-to-one correspondence between the candidate preference optimization loss and the negative sample speech feature; each negative sample generates one candidate preference optimization loss. The candidate preference optimization loss can be a loss value calculated based on the DPO algorithm or other preference learning algorithms. For example, if three negative sample speech features are generated using different sampling seeds for the same positive sample, three candidate preference optimization losses will be calculated separately.

[0083] In practical applications, there are various ways to calculate the mean of multiple candidate preference optimization losses to obtain the preference optimization loss. The specific method chosen depends on the actual situation, and this specification does not limit this approach. One possible implementation in this specification is to take the arithmetic mean of all candidate preference optimization losses as the preference optimization loss for this iteration. Another possible implementation is to perform a weighted average of the candidate preference optimization losses, where the weights are determined based on the similarity between the speech features of each negative sample and the speech features of the positive sample. This ensures that negative samples closer to the positive samples have a larger weight in the loss, thereby guiding the model to prioritize distinguishing difficult-to-distinguish negative samples.

[0084] For example, suppose that during preference optimization training, the reference speech synthesis model is called three times using different sampling seeds for the same positive sample speech feature, generating three different negative sample speech features. The reference speech features and the target text features are concatenated as conditional input features, which are then input into the initial speech synthesis model and the reference speech synthesis model, respectively, to calculate the first and second positive probability information for the positive sample speech features. For each of the three negative sample speech features, the first and second negative probability information are calculated, resulting in three sets of probability information combinations. A candidate preference optimization loss is calculated for each set of probability information, and finally, the average of the three candidate preference optimization losses is taken to obtain the preference optimization loss for this iteration.

[0085] The scheme implemented in this specification constructs multiple negative sample speech features for the same positive sample, enabling the preference signal to cover various possibilities of spontaneous model output. By independently calculating the candidate preference optimization loss for each negative sample, it ensures that the preference information of each negative sample can be measured individually. By averaging the optimization losses of multiple candidate preferences, the one-sidedness of the preference signal caused by a single negative sample is eliminated, enhancing the stability and representativeness of the preference loss estimation and improving the robustness of model training.

[0086] In one optional embodiment of this specification, positive sample speech features include a plurality of positive sample speech sub-features arranged in sequence, and negative sample speech features include a plurality of negative sample speech sub-features arranged in sequence. The above-mentioned determination of the first positive probability information of positive sample speech features and the first negative probability information of negative sample speech features under the initial speech synthesis model based on conditional input features may include the following steps: Based on the conditional input features, the first prediction probability corresponding to each positive sample speech sub-feature is determined sequentially according to the arrangement order of multiple positive sample speech sub-features, and the first positive probability information is determined based on multiple first prediction probabilities. Based on the conditional input features, the second prediction probability corresponding to each negative sample speech sub-feature is determined sequentially according to the arrangement order of multiple negative sample speech sub-features, and the first negative probability information is determined based on multiple second prediction probabilities.

[0087] It should be noted that a positive sample speech sub-feature refers to a single discrete speech marker arranged in chronological order within a positive sample speech feature sequence; it is the constituent unit of the positive sample speech feature. A positive sample speech sub-feature can be a single marker in a discrete acoustic marker sequence obtained after encoding a target speech segment using a speech encoder. Each positive sample speech sub-feature corresponds to the true acoustic features of the target speech segment at a specific moment. For example, if the target speech segment is encoded by a speech encoder to obtain 600 sequentially arranged discrete speech markers, each marker constitutes a positive sample speech sub-feature.

[0088] Negative sample speech sub-features refer to individual discrete speech tokens arranged in chronological order within a negative sample speech feature sequence; they are the building blocks of negative sample speech features. A negative sample speech sub-feature can be a single token in a sequence of discrete acoustic tokens generated autoregressively by a reference speech synthesis model. Each negative sample speech sub-feature corresponds to the acoustic features at a specific moment in the generation process. For example, 600 sequentially arranged discrete speech tokens generated autoregressively by a reference speech synthesis model can each constitute a negative sample speech sub-feature.

[0089] The first prediction probability refers to the prediction probability of the current positive sample speech sub-feature by the initial speech synthesis model, given the conditional input features and previously confirmed positive sample speech sub-features. The first prediction probability can be the conditional probability output by the model at each position during stepwise prediction in an autoregressive manner, or it can be a local conditional probability calculated using methods such as sliding windows. The first prediction probability is used to quantify the degree of matching between the model and the real target speech at each position. For example, for the t-th positive sample speech sub-feature in the positive sample speech feature sequence, the initial speech synthesis model predicts the conditional probability of the t-th positive sample speech sub-feature based on the conditional input features and the previous t-1 positive sample speech sub-features; this is the first prediction probability corresponding to that position.

[0090] The second prediction probability refers to the prediction probability of the current negative sample speech sub-feature by the initial speech synthesis model, given the conditional input features and previously confirmed negative sample speech sub-features. The second prediction probability can be the conditional probability output by the model at each position during stepwise prediction in an autoregressive manner, or it can be a local conditional probability calculated using methods such as sliding windows. The second prediction probability is used to quantify the degree of matching between the model and spontaneous style speech at each position. For example, for the t-th negative sample speech sub-feature in the negative sample speech feature sequence, the initial speech synthesis model predicts the conditional probability of the t-th negative sample speech sub-feature based on the conditional input features and the previous t-1 negative sample speech sub-features; this is the second prediction probability corresponding to that position.

[0091] For example, suppose the positive sample speech features are a sequence of 600 sequentially arranged discrete speech tokens, and the negative sample speech features are also a sequence of 600 sequentially arranged discrete speech tokens. After inputting the conditional input features into the initial speech synthesis model, the model predicts the first prediction probability at each time step based on the conditional input features and the previously confirmed positive sample speech sub-features, according to the order of the positive sample speech sub-features. After traversing all 600 positions, 600 first prediction probabilities are obtained. The logarithm of the conditional probability at each position is taken and summed to obtain the first positive probability information. Similarly, the model predicts the second prediction probability at each position sequentially using the same autoregressive method, according to the order of the negative sample speech sub-features. After traversing all 600 positions, the logarithm of the conditional probability at each position is taken and summed to obtain the first negative probability information.

[0092] The scheme implemented in this specification refines the granularity of sequence-level probability calculation to the individual label level by splitting positive and negative sample speech features into sequentially arranged sub-feature sequences. By calculating the predicted probability for each sub-feature in order, the conditional probability at each position fully utilizes the previously confirmed sub-feature information, ensuring that the probability estimation per token is consistent with the generation mechanism of the autoregressive model. By determining the sequence-level probability information based on multiple predicted probabilities, an accurate assessment of the joint probability of the entire speech feature sequence is achieved, providing an accurate probabilistic basis for the subsequent calculation of preference optimization loss.

[0093] In one optional embodiment of this specification, the initial speech synthesis model includes an acoustic synthesis unit and an acoustic decoding unit. The above-described parameter adjustment of the initial speech synthesis model based on preference optimization loss to obtain the trained speech synthesis model may include the following steps: The parameters of the acoustic synthesis unit are adjusted based on the preference optimization loss, while the parameters of the acoustic decoding unit remain unchanged, to obtain the trained speech synthesis model.

[0094] It should be noted that the acoustic synthesis unit (ASU) is the module in the initial speech synthesis model responsible for mapping conditional input features into discrete speech feature sequences. The ASU can be a pre-trained autoregressive language model whose parameters are updated via gradient backpropagation during preference optimization training; or it can be a sequence-to-sequence model based on the Transformer architecture, used as a policy model for parameter adjustment during preference optimization. The ASU is used to generate speech feature sequences consistent with the reference style given reference conditions. For example, the ASU might be a large language model currently being trained, receiving concatenated reference speech features and text features, and generating a speech feature sequence in an autoregressive manner.

[0095] An acoustic decoding unit (CDU) is the module in the initial speech synthesis model responsible for reconstructing an audio waveform from a speech feature sequence combined with timbre information. The CDU can be a CFM-based acoustic decoder, whose parameters are frozen during preference optimization training, or it can be based on a vocoder or other waveform reconstruction module, not participating in parameter updates during preference optimization. The CDU is used to reconstruct the discrete speech features output by the acoustic synthesis unit into a continuous audio waveform with high quality. For example, the CDU receives the speech feature sequence output by the acoustic synthesis unit and a timbre vector extracted from a reference audio, and generates the final audio waveform through conditional flow matching.

[0096] For example, assume the initial speech synthesis model includes a large language model as the acoustic synthesis unit and a conditional flow matching decoder as the acoustic decoding unit. During preference optimization training, conditional input features are input to the acoustic synthesis unit to generate a predicted speech feature sequence. When updating parameters based on preference optimization loss, gradients are only propagated back to the acoustic synthesis unit to update its model parameters; the parameters of the acoustic decoding unit remain frozen throughout the training process and do not participate in gradient calculation. Preference optimization training is performed entirely at the speech feature level, and the acoustic decoding unit always uses the parameters learned during pre-training to reconstruct the speech feature sequence output by the acoustic synthesis unit into an audio waveform. After multiple iterations, the acoustic synthesis unit learns to generate speech features consistent with the reference style under given reference conditions, and the timbre control capability of the acoustic decoding unit remains unaffected, resulting in a trained speech synthesis model.

[0097] The scheme implemented in this specification achieves modular separation of model functions by dividing the initial speech synthesis model into acoustic synthesis units and acoustic decoding units. By optimizing the loss according to preferences and adjusting only the parameters of the acoustic synthesis units, the enhancement of style control capabilities is concentrated in the module responsible for feature generation. By keeping the parameters of the acoustic decoding units unchanged, the module responsible for waveform reconstruction always uses the parameters learned in the pre-training stage, avoiding the problem that fine-tuning all parameters may destroy the learned timbre reconstruction capabilities.

[0098] The scheme implemented in this specification ensures consistency between the two speech segments in terms of speaker identity and style background by acquiring sample speech containing both a reference speech segment and a target speech segment. The reference and target speech segments are encoded separately to obtain reference speech features and positive sample speech features, ensuring the model can establish an effective mapping from reference conditions to the target output. The target speech text is encoded to obtain target text features, and the reference speech synthesis model is invoked to generate negative sample speech features based on the target text features, providing a benchmark for positive and negative sample comparison. Based on the reference speech features, target text features, positive sample speech features, and negative sample speech features, the parameters of the initial speech synthesis model are adjusted, constructing a preference constraint mechanism for positive and negative sample comparison. This allows the model to simultaneously obtain optimization signals in both directions—moving closer to positive samples and moving away from negative samples—during parameter updates, thereby strengthening the model's ability to follow reference audio conditions, overcoming the limitations of supervision by a single real target sequence, and improving style consistency between synthesized speech and reference audio.

[0099] In one optional embodiment of this specification, the above-described speech synthesis model training method may further include the following steps: During the parameter adjustment of the initial speech synthesis model, the parameter-adjusted initial speech synthesis model is called, and the first verification speech feature is generated based on the verification reference speech features and the verification target text features. The initial speech synthesis model with adjusted parameters is invoked, and a second verification speech feature is generated based on the verification target text features. Determine the path similarity between the first and second verification speech features; If the path similarity is less than a preset similarity threshold, stop adjusting the parameters of the initial speech synthesis model and obtain the trained speech synthesis model.

[0100] It should be noted that validation reference speech features refer to the reference speech features used for monitoring and evaluation during training. These features are derived from encoded reference speech segments in the validation set. Validation reference speech features can be discrete speech tag sequences obtained by encoding reference speech segments selected from an independent validation set, or reference speech features obtained from a validation subset partitioned from the training set. Validation reference speech features are used to provide style reference conditions for the model during evaluation. For example, reference speech segments are selected from the validation set and encoded using a speech encoder to obtain validation reference speech features.

[0101] Validation target text features refer to the features obtained by encoding the text corresponding to the target speech segment in the validation set. Validation target text features can be discrete text tag sequences obtained by encoding the text corresponding to the target speech segment in the validation set using a text encoder. Validation target text features are used to provide semantic conditions for the model to generate speech. For example, the validation target text features are obtained by encoding the text corresponding to the target speech segment in the validation set using a text encoder.

[0102] The first validation speech feature refers to the speech feature generated by the parameter-adjusted initial speech synthesis model under the condition of concatenating validation reference speech features and validation target text features. The first validation speech feature can be a discrete speech tag sequence. The first validation speech feature reflects the generation result of the model when reference conditions are available. For example, the first validation speech feature is the speech feature sequence generated by the model in an autoregressive manner after concatenating the validation reference speech features and validation target text features and inputting them into the currently trained model.

[0103] The second validation speech feature refers to the speech feature generated by the parameter-adjusted initial speech synthesis model under the condition that only the validation target text feature is input, without concatenating the validation reference speech feature. The second validation speech feature can be a discrete speech tag sequence. The second validation speech feature reflects the spontaneous output of the model without reference conditions. For example, if only the validation target text feature is input into the parameter-adjusted initial speech synthesis model, the speech feature sequence generated by the model in an autoregressive manner is the second validation speech feature.

[0104] Path similarity is a measure of the degree of similarity between the first and second verification speech features. Path similarity can be calculated based on a sequence distance metric, such as edit distance, hidden layer cosine distance, or dynamic time warping distance. A lower path similarity value indicates a greater difference in output between conditions with and without reference, meaning a higher degree of adherence to the reference conditions. For example, the path similarity can be obtained by measuring the sequence distance between the first and second verification speech features using edit distance and then normalizing it.

[0105] A preset similarity threshold is a critical similarity value used to determine whether a model has fully learned to utilize reference conditions. The preset similarity threshold can be a fixed value set empirically or an adaptive value dynamically determined based on validation set statistics. When the path similarity is less than the preset similarity threshold, it indicates that the difference between the model's output with and without reference conditions has reached the expected level, and training can be stopped. For example, if the preset similarity threshold is set to 0.3, when the path similarity drops below 0.3, it is considered that the model has learned to adjust its generation style according to reference conditions.

[0106] For example, assume that during preference optimization training, evaluation is performed on the validation set every 100 iterations. The first validation speech feature is generated by concatenating the validation reference speech features and the validation target text features into an initial speech synthesis model with adjusted parameters; the second validation speech feature is generated by inputting only the validation target text features into the initial speech synthesis model with adjusted parameters. The sequence distance between the two is measured by edit distance, and the path similarity is obtained after normalization. In the early stages of training, the model has not yet learned to utilize the reference conditions, and the output difference between the two conditions is small, resulting in high path similarity. As training progresses, the model gradually learns to adjust the generated results according to the reference conditions, the output difference between the two conditions increases, and the path similarity gradually decreases. When the path similarity is less than a preset similarity threshold, it indicates that the model has fully learned to utilize the reference speech conditions, parameter adjustment stops, and the trained speech synthesis model is obtained.

[0107] By applying the scheme of the embodiments in this specification, a first verification speech feature and a second verification speech feature are generated by calling the initial speech synthesis model after parameter adjustment during the parameter adjustment process, thereby achieving a comparative evaluation of the model's output behavior under reference conditions and without reference conditions. The degree to which the model follows the reference conditions is quantified by determining the path similarity between the first and second verification speech features. Automated convergence judgment based on model behavior differences is achieved by stopping parameter adjustment when the path similarity is less than a preset similarity threshold, avoiding overtraining or premature termination and ensuring the model training effect.

[0108] An example, Figure 2 This diagram illustrates a process of training a speech synthesis model according to an embodiment of this specification. Figure 2 As shown, the first step is data filtering. 15 to 30 seconds of real speech are selected as sample speech. The sample speech includes a reference speech segment and a target speech segment.

[0109] The sample speech is segmented and encoded. The reference speech segment of the first 5 seconds of the sample speech is input into the speech encoder to generate reference speech features P=[P_1, P_2, ..., P_m]; the target speech text corresponding to the target speech segment is encoded to obtain target text features T=[T_1, T_2, ..., T_k]; the real speech corresponding to the target speech segment in the sample speech is input into the speech encoder to obtain positive sample speech features yw=[yw_1, yw_2, ..., yw_n].

[0110] Offline generation of negative examples. Without concatenating the reference speech features, the target text features T are input into the reference speech synthesis model π_ref, which generates negative sample speech features yl=[yl_1, yl_2, ..., yl_n] based on the target text features T via autoregression. The reference speech synthesis model π_ref can be called multiple times based on different sampling seeds to generate multiple negative sample speech features. The mean of the candidate preference optimization loss corresponding to the multiple negative sample speech features of the same positive example is calculated separately to increase the diversity of negative samples, as shown in formula (1): (1) in, This represents the preference optimization loss in the case of multiple negative examples; K represents the total number of negative samples, ranging from 1 to 5; k represents the k-th negative sample, ranging from 1 to K. θ represents the preference optimization loss for a single negative sample; θ represents the parameters of the initial speech synthesis model in the current training. Indicates positive sample speech features. Let represent the speech feature of the k-th negative sample. By using the mechanism of averaging multiple negative samples, the preference signal can cover various possibilities of the model's spontaneous output, enhancing the representativeness and stability of the preference optimization loss and avoiding the one-sidedness of the preference signal caused by a single negative sample.

[0111] Construct preference pairs by assembling the reference speech feature P, the target text feature T, the positive sample speech feature yw, and the negative sample speech feature yl.

[0112] DPO forward inference obtains four types of probabilistic information through an initial speech synthesis model and a reference speech synthesis model. The reference speech feature P and the target text feature T are concatenated into a conditional input feature [P;T], which is then input into the initial speech synthesis model π_θ and the reference speech synthesis model π_ref, respectively. Based on the conditional input feature [P;T], the initial speech synthesis model π_θ determines the first positive probability information of the positive sample speech feature yw and the first negative probability information of the negative sample speech feature yl; the reference speech synthesis model π_ref, based on the conditional input feature [P;T], determines the second positive probability information of the positive sample speech feature yw and the second negative probability information of the negative sample speech feature yl.

[0113] The initial speech synthesis model π_θ, based on conditional input features [P;T], determines the first positive probability information of the positive sample speech features yw. As shown in formula (2): (2) Where n is the length of the speech feature sequence, and the input of LLM is [P;T], which predicts the speech features at each position step by step in an autoregressive manner.

[0114] The initial speech synthesis model π_θ, based on conditional input features [P;T], determines the first negative probability information of the negative sample speech feature yl. As shown in formula (3): (3) The reference speech synthesis model π_ref determines the second positive probability information of the positive sample speech feature yw based on the conditional input features [P;T]. As shown in formula (4): (4) The reference speech synthesis model π_ref determines the second negative probability information of the negative sample speech feature yl based on the conditional input features [P;T]. As shown in formula (5): (5) Calculate the preference optimization loss. Based on the first positive probability information, the first negative probability information, the second positive probability information, and the second negative probability information, determine the preference optimization loss. As shown in formula (6): (6) in, This represents the sigmoid function. is the temperature coefficient, with a value ranging from 0.1 to 0.5. A is shown in formula (7): (7) Parameter tuning only updates the initial speech synthesis model π_θ. Based on preference-based loss optimization, the parameters of the acoustic synthesis units in the initial speech synthesis model π_θ are adjusted, while the parameters of the acoustic decoding units remain unchanged. The parameters of the reference speech synthesis model π_ref are frozen during training and do not participate in gradient updates.

[0115] Monitoring and evaluation are conducted, and the path similarity index S is calculated. During the parameter adjustment of the initial speech synthesis model π_θ, the parameter-adjusted initial speech synthesis model π_θ is periodically called. Based on the verification reference speech feature P_val and the verification target text feature T_val, a first verification speech feature Xcond is generated, and based on the verification target text feature T_val, a second verification speech feature Xuncond is generated. The path similarity between the first verification speech feature Xcond and the second verification speech feature Xuncond is determined.

[0116] The path similarity index S between the first verification speech feature Xcond and the second verification speech feature Xuncond is determined as shown in formula (8): (8) Where D(Xcond,Xuncond) is the sequence distance metric function, which can be implemented as follows: Edit distance: treating the two speech feature sequences as discrete symbol strings and calculating the minimum number of edit operations; Hidden layer cosine distance: taking the mean vectors of the two sequences in the hidden layer of the model and calculating the cosine distance; Dynamic time warping distance: handling the optimal alignment distance when the two sequences have unequal lengths. Dmax is a normalization constant, determined statistically by the validation set, used to map D to the [0,1] interval. The lower the S value, the higher the degree of utilization of the reference audio conditions by the model.

[0117] Convergence determination. If the path similarity S is not less than the preset similarity threshold τ, it indicates that convergence has not occurred, and DPO forward inference, preference optimization loss calculation, and parameter adjustment continue. If the path similarity S is less than the preset similarity threshold τ, it indicates that convergence has occurred, and parameter adjustment of the initial speech synthesis model is stopped, resulting in a trained speech synthesis model. Here, τ is 0.3–0.5.

[0118] Corresponding to the above-described embodiments of the speech synthesis model training method, this specification also provides a method for constructing speech synthesis training samples. Figure 3 A flowchart illustrating a method for constructing training samples for speech synthesis according to an embodiment of this specification is shown, such as... Figure 3 As shown, the following steps may be included: Step 302: Obtain sample speech, wherein the sample speech includes a reference speech segment and a target speech segment; Step 304: Encode the reference speech segment to obtain reference speech features, and encode the target speech segment to obtain positive sample speech features; Step 306: Encode the target speech text corresponding to the target speech segment to obtain the target text features, and call the reference speech synthesis model to generate negative sample speech features based on the target text features; Step 308: Construct speech synthesis training samples based on reference speech features, target text features, positive sample speech features, and negative sample speech features.

[0119] It should be noted that speech synthesis training samples refer to structured data units composed of reference speech features, target text features, positive sample speech features, and negative sample speech features, used for preference optimization training. Each speech synthesis training sample contains a preference pair, where positive sample speech features are positive examples and negative sample speech features are negative examples. For example, a speech synthesis training sample contains: reference speech features, target text features, positive sample speech features, and negative sample speech features, which together constitute a complete preference pair training unit.

[0120] The scheme implemented in this specification provides a unified source of speech data for positive examples and conditional inputs of preference pairs by acquiring sample speech containing reference and target speech segments. Reference speech features are obtained by encoding the reference speech segment, and positive sample speech features are obtained by encoding the target speech segment, thus obtaining style conditional inputs and training target benchmarks. Target text features are obtained by encoding the target speech text, and negative sample speech features are generated based on the target text features by calling the reference speech synthesis model, obtaining the model's spontaneous prosodic output in the absence of style references as a comparison benchmark. By constructing speech synthesis training samples based on reference speech features, target text features, positive sample speech features, and negative sample speech features, data construction and model training are decoupled, allowing subsequent preference optimization training to directly read pre-constructed preference pair data, improving the flexibility and efficiency of the training process.

[0121] The above is an illustrative scheme of a speech synthesis training sample construction method according to this embodiment. It should be noted that the technical solution of this speech synthesis training sample construction method belongs to the same concept as the technical solution of the speech synthesis model training method described above. For details not described in detail in the technical solution of the speech synthesis training sample construction method, please refer to the description of the technical solution of the speech synthesis model training method described above.

[0122] See Figure 4 , Figure 4 This specification shows a flowchart of a speech synthesis method according to an embodiment, which may specifically include the following steps: Step 402: Obtain the target reference audio and the text to be synthesized; Step 404: Encode the target reference audio to obtain reference speech features, and encode the text to be synthesized to obtain text features; Step 406: Call the speech synthesis model to generate the target synthesized speech based on the reference speech features and the text features to be synthesized. The speech synthesis model is trained using the speech synthesis model training method.

[0123] It's important to note that the target reference audio refers to the input audio provided by the user to specify the speaking style of the synthesized speech. The target reference audio can be a speech clip recorded by the user in real-time or an existing speech clip selected from an audio library. Its length and content are unrestricted, as long as it contains sufficient stylistic information about the speaker. The target reference audio provides stylistic and prosodic reference conditions for speech synthesis, enabling the generated synthesized speech to mimic the speaker's timbre and prosodic characteristics in the target reference audio. For example, a user might provide a 5-second speech clip as the target reference audio, which contains information about the speaker's timbre, prosodicity, and speaking style.

[0124] The text to be synthesized refers to the text content input by the user that needs to be synthesized into speech. The text to be synthesized can be any natural language text, such as a paragraph, an article, or the content of a sentence. The text to be synthesized provides semantic conditions for speech synthesis, determining the specific linguistic content of the synthesized speech. For example, the user inputs "The weather is really nice today, perfect for a walk" as the text to be synthesized.

[0125] The text features to be synthesized refer to the feature representation obtained after encoding the text to be synthesized. They are the semantic information of the text that the model needs to rely on when generating synthesized speech. The text features to be synthesized can be a discrete text tag sequence obtained by encoding the text to be synthesized by a text encoder. For example, the text to be synthesized is encoded into a text tag sequence by a text encoder, which serves as the semantic conditional input for the speech synthesis model to generate speech.

[0126] A speech synthesis model refers to a model trained using the aforementioned speech synthesis model training methods. During the training phase, it improves the stylistic consistency between the synthesized speech and the reference audio through preference optimization. The speech synthesis model includes an acoustic synthesis unit and an acoustic decoding unit. During the inference phase, the parameters of the speech synthesis model are the final parameters after training.

[0127] Target synthesized speech refers to the synthesized speech output generated by a speech synthesis model based on the features of a reference speech and the features of the text to be synthesized. The target synthesized speech should have a similar speaking style, timbre, and prosodic features to the target reference audio, while also containing the semantic content described by the text to be synthesized. For example, if the target reference audio is a segment of a user's speech, then the target synthesized speech is the speech of that user speaking the content of the text to be synthesized.

[0128] For example, in a learning assistance scenario, students can provide a short audio clip of their instructor as a target reference audio. The system, based on the reference speech features of this audio clip and the synthesized text features of the course text, generates a target synthesized speech that matches the instructor's speaking style. This allows students to review course content after class with a voice style similar to the original classroom, reducing the learning adaptation cost caused by changes in the sound environment and improving learning continuity. In an audiobook reading scenario, users can upload a short audio clip of their own voice as a target reference audio. The system, based on the reference speech features of this audio clip and the synthesized text features of the course text, calls a speech synthesis model to generate a target synthesized speech with the user's speaking style. This allows users to read book content in their own voice style without having to record the entire content themselves.

[0129] The scheme implemented in this specification uses a speech synthesis model trained based on preference optimization to fuse the features of the reference speech and the text to be synthesized to generate the target synthesized speech. This allows the synthesized speech to accurately express the semantic content of the text to be synthesized while maintaining the speaking style of the target reference audio, thus achieving speech synthesis with high style consistency.

[0130] The above is an illustrative scheme of a speech synthesis method according to this embodiment. It should be noted that the technical solution of this speech synthesis method belongs to the same concept as the technical solution of the speech synthesis model training method described above. For details not described in detail in the technical solution of the speech synthesis method, please refer to the description of the technical solution of the speech synthesis model training method described above.

[0131] Corresponding to the above-described speech synthesis method embodiments, this specification also provides speech synthesis model embodiments, which may include an embedding layer, a language model backbone network, an acoustic synthesis unit, and an acoustic decoding unit connected in sequence.

[0132] The embedding layer receives conditional input features and maps discrete speech and text tokens to a continuous high-dimensional vector space through vector mapping operations, outputting the corresponding embedding vector sequence, which is then transmitted to the language model backbone network. The conditional input features include reference speech features and the text features to be synthesized.

[0133] The embedding layer generates a style reference embedding vector for each discrete speech tag in the reference speech features and a semantic embedding vector for each discrete text tag in the text features to be synthesized. The style reference embedding vector and the semantic embedding vector are then combined in the order of concatenation to form an embedding vector sequence. The dimension of the embedding vectors can be set according to the actual configuration requirements of the model.

[0134] The language model backbone network receives the embedded vector sequence and models the contextual relationships between each position in the embedded vector sequence through a multi-layer self-attention mechanism and a feedforward network. It outputs the corresponding positional hidden state sequence and transmits the positional hidden state sequence to the acoustic synthesis unit.

[0135] The language model backbone network can include multiple stacked Transformer decoder layers, each of which includes a multi-head self-attention sublayer and a feedforward neural network sublayer. Through layer-by-layer processing of the multiple Transformer decoder layers, the language model backbone network can fully capture the interaction between the speaking style information carried in the reference speech features and the semantic information carried in the text features to be synthesized, and output a positional hidden state sequence that integrates style and semantic conditions.

[0136] The acoustic synthesis unit receives the positional hidden state sequence output by the language model backbone network and, through autoregressive prediction, progressively predicts the discrete speech tokens corresponding to each position according to the order of the target speech features, outputting the corresponding speech feature sequence, which is then transmitted to the acoustic decoding unit. The acoustic synthesis unit may include a linear mapping layer and an activation function layer. The linear mapping layer maps the positional hidden states to a preset speech token category space, outputting the predicted probability for each candidate speech token category. The activation function layer determines the discrete speech token corresponding to each position based on the predicted probability. Following an autoregressive approach, the acoustic synthesis unit uses the discrete speech tokens predicted at each step as input for the next step, progressively generating a complete speech feature sequence.

[0137] The acoustic decoding unit receives the speech feature sequence output by the acoustic synthesis unit and, through conditional flow matching, combines the speech feature sequence with timbre vectors to reconstruct an audio waveform signal, outputting the corresponding synthesized speech. The acoustic decoding unit can be a conditional flow matching model, using the timbre vector of the reference audio to guide timbre reconstruction during the decoding process, ensuring that the generated synthesized speech maintains timbre consistency with the reference audio. The parameters of the acoustic decoding unit remain unchanged during both the training and inference phases, preserving its learned timbre reconstruction capabilities during training.

[0138] The above is an illustrative scheme of a speech synthesis model according to this embodiment. It should be noted that the technical solution of this speech synthesis model belongs to the same concept as the technical solutions of the speech synthesis model training method and the speech synthesis method described above. The speech synthesis model serves as the execution carrier of the above methods, and is used to implement the core steps of inputting reference speech features and text features into the model and outputting synthesized speech. For details not described in detail in this speech synthesis model technical solution, please refer to the description of the technical solutions of the speech synthesis model training method and the speech synthesis method described above.

[0139] Considering the large number of parameters in speech synthesis models and the limited computing resources on the client side, the speech synthesis method proposed in the embodiments of this specification can be applied to, for example... Figure 5 The speech synthesis system shown is not limited to this. See also Figure 5 , Figure 5 This specification illustrates an architecture diagram of a speech synthesis system provided in one embodiment. The speech synthesis system may include a client 502 and a server 504.

[0140] Client 502 is used to send the target reference audio and the text to be synthesized to server 504.

[0141] The server returns a 504 error and is used to obtain the target reference audio and the text to be synthesized; the target reference audio is encoded to obtain reference speech features, and the text to be synthesized is encoded to obtain text features; the speech synthesis model is called to generate the target synthesized speech based on the reference speech features and the text features to be synthesized, wherein the speech synthesis model is trained based on the speech synthesis model training method described above.

[0142] Client 502 is also used to receive the target synthesized speech sent by server 504.

[0143] like Figure 5 As shown, the speech synthesis model is deployed in server 504. Server 504 can connect to one or more clients 502 via a local area network (LAN), wide area network (WAN), Internet, or other types of data network. Client 502 may include, but is not limited to, smartphones, tablets, laptops, PDAs, personal computers, smart home devices, and in-vehicle devices. Client 502 can also interact with users through a graphical user interface to invoke the speech synthesis model, thereby implementing the speech synthesis method provided in the embodiments of this specification.

[0144] It is worth noting that the speech synthesis method provided in the embodiments of this specification is generally executed by the server. However, in other embodiments of this specification, if the client's operating resources can meet the deployment and operating conditions of the speech synthesis model, the client can also have similar functions to the server, thereby executing the speech synthesis method provided in the embodiments of this specification.

[0145] Corresponding to the above-described embodiments of the speech synthesis model training method, this specification also provides embodiments of the speech synthesis model training device. Figure 6a A schematic diagram of a speech synthesis model training device according to one embodiment of this specification is shown. Figure 6a As shown, the device includes: The first acquisition module 602 is configured to acquire sample speech, wherein the sample speech includes a reference speech segment and a target speech segment.

[0146] The first encoding module 604 is configured to encode a reference speech segment to obtain reference speech features, and to encode a target speech segment to obtain positive sample speech features.

[0147] The second encoding module 606 is configured to encode the target speech text corresponding to the target speech segment, obtain the target text features, and call the reference speech synthesis model to generate negative sample speech features based on the target text features.

[0148] The adjustment module 608 is configured to adjust the parameters of the initial speech synthesis model based on reference speech features, target text features, positive sample speech features, and negative sample speech features to obtain a trained speech synthesis model.

[0149] Optionally, the adjustment module 608 is further configured to generate conditional input features based on reference speech features and target text features; based on the conditional input features, determine the first positive probability information of positive sample speech features under the initial speech synthesis model and the first negative probability information of negative sample speech features under the initial speech synthesis model; based on the conditional input features, determine the second positive probability information of positive sample speech features under the reference speech synthesis model and the second negative probability information of negative sample speech features under the reference speech synthesis model; determine the preference optimization loss according to the first positive probability information, the first negative probability information, the second positive probability information, and the second negative probability information; and adjust the parameters of the initial speech synthesis model according to the preference optimization loss to obtain the trained speech synthesis model.

[0150] Optionally, the adjustment module 608 is further configured to compare the first positive probability information and the first negative probability information to determine the first probability difference; compare the second positive probability information and the second negative probability information to determine the second probability difference; and determine the preference optimization loss based on the first probability difference and the second probability difference.

[0151] Optionally, the number of negative sample speech features is multiple. The adjustment module 608 is further configured to determine multiple candidate preference optimization losses based on the first positive probability information, the first negative probability information, the second positive probability information, and the second negative probability information corresponding to each negative sample speech feature, wherein the candidate preference optimization loss corresponds one-to-one with the negative sample speech feature; and to calculate the average of the multiple candidate preference optimization losses to obtain the preference optimization loss.

[0152] Optionally, the positive sample speech features include multiple positive sample speech sub-features arranged in sequence, and the negative sample speech features include multiple negative sample speech sub-features arranged in sequence. The adjustment module 608 is further configured to, based on the conditional input features, sequentially determine the first prediction probability corresponding to each positive sample speech sub-feature according to the arrangement order of the multiple positive sample speech sub-features, and determine first positive probability information based on the multiple first prediction probabilities; based on the conditional input features, sequentially determine the second prediction probability corresponding to each negative sample speech sub-feature according to the arrangement order of the multiple negative sample speech sub-features, and determine first negative probability information based on the multiple second prediction probabilities.

[0153] Optionally, the initial speech synthesis model includes an acoustic synthesis unit and an acoustic decoding unit; the adjustment module 608 is further configured to optimize the loss according to preferences, adjust the parameters of the acoustic synthesis unit, and keep the parameters of the acoustic decoding unit unchanged, so as to obtain the trained speech synthesis model.

[0154] Optionally, the first encoding module 604 is further configured to invoke the same speech encoder to encode the reference speech segment and the target speech segment respectively, so as to obtain the reference speech features and the positive sample speech features.

[0155] Optionally, the device further includes: a verification module configured to, during the parameter adjustment process of the initial speech synthesis model, call the parameter-adjusted initial speech synthesis model, generate a first verification speech feature based on verification reference speech features and verification target text features; call the parameter-adjusted initial speech synthesis model, generate a second verification speech feature based on verification target text features; determine the path similarity between the first verification speech feature and the second verification speech feature; and, if the path similarity is less than a preset similarity threshold, stop adjusting the parameters of the initial speech synthesis model to obtain a trained speech synthesis model.

[0156] The above is an illustrative scheme of a speech synthesis model training device according to this embodiment. It should be noted that the technical solution of this speech synthesis model training device and the technical solution of the speech synthesis model training method described above belong to the same concept. For details not described in detail in the technical solution of the speech synthesis model training device, please refer to the description of the technical solution of the speech synthesis model training method described above.

[0157] In one optional embodiment of this specification, see [link to embodiment]. Figure 6b , Figure 6b A schematic diagram of another speech synthesis model training device provided in one embodiment of this specification is shown. Figure 6b As shown, the speech synthesis model training device includes: a first acquisition module 602, a first encoding module 604, a second encoding module 606, an adjustment module 608, and a CFM decoding module 610.

[0158] The first acquisition module 602 is used to acquire 15 to 30 seconds of sample speech and segment the sample speech. The first segment of speech is used as a reference speech segment, and the second segment of speech is used as a target speech segment. The reference speech segment and the target speech segment are then transmitted to the first encoding module 604.

[0159] The first encoding module 604 encodes the segmented reference speech segment into the speech encoder to obtain reference speech features, and encodes the target speech segment into the speech encoder to obtain positive sample speech features. Finally, the reference speech features and positive sample speech features are transmitted to the adjustment module 608.

[0160] The second encoding module 606 encodes the target speech text corresponding to the target speech segment to obtain target text features. Without concatenating the reference speech features, only the target text features are input into the reference speech synthesis model. The reference speech synthesis model then generates negative sample speech features based on the target text features through autoregression. Finally, the target text features and negative sample speech features are transmitted to the adjustment module 608.

[0161] The adjustment module 608 concatenates the reference speech features and the target text features into conditional input features. These conditional input features are then input into both the reference speech synthesis model and the initial speech synthesis model. The two models calculate probability information for positive and negative sample speech features, respectively, and the preference optimization loss is obtained through DPO preference loss calculation.

[0162] The adjustment module adjusts the parameters of the initial speech synthesis model based on the preference optimization loss, updating its parameters. The parameters of the reference speech synthesis model are frozen during training and do not participate in gradient updates. After training is complete, the parameters of the initial speech synthesis model are determined as the final parameters.

[0163] The speech feature sequence output by the acoustic synthesis unit in the trained speech synthesis model is transmitted to the CFM decoding module 610. The CFM decoding module 610 receives the speech feature sequence and extracts the timbre vector of the reference speech segment. The CFM decoding module 610 then reconstructs the audio waveform signal from the speech feature sequence and timbre vector using conditional flow matching, outputting the synthesized speech. The parameters of the CFM decoding module 610 remain unchanged during the training phase, preserving its learned timbre reconstruction capability, ensuring that the final output synthesized speech maintains timbre consistency with the reference speech segment.

[0164] Corresponding to the above-described embodiments of the speech synthesis method, this specification also provides embodiments of the speech synthesis apparatus. Figure 7 A schematic diagram of a speech synthesis device according to one embodiment of this specification is shown. Figure 7 As shown, the device includes: The second acquisition module 702 is configured to acquire the target reference audio and the text to be synthesized.

[0165] The third encoding module 704 is configured to encode the target reference audio to obtain reference speech features, and to encode the text to be synthesized to obtain text features.

[0166] The generation module 706 is configured to call the speech synthesis model to generate target synthesized speech based on reference speech features and text features to be synthesized. The speech synthesis model is trained using the speech synthesis model training method described above.

[0167] The above is an illustrative scheme of a speech synthesis device according to this embodiment. It should be noted that the technical solution of this speech synthesis device and the technical solution of the above-described speech synthesis method belong to the same concept. For details not described in detail in the technical solution of the speech synthesis device, please refer to the description of the technical solution of the above-described speech synthesis method.

[0168] Corresponding to the above embodiments of the speech synthesis training sample construction method, this specification also provides embodiments of the speech synthesis training sample construction apparatus. Figure 8 A schematic diagram of a speech synthesis training sample construction device according to one embodiment of this specification is shown. Figure 8 As shown, the device includes: The third acquisition module 802 is configured to acquire sample speech, wherein the sample speech includes a reference speech segment and a target speech segment.

[0169] The fourth encoding module 804 is configured to encode the reference speech segment to obtain reference speech features, and to encode the target speech segment to obtain positive sample speech features.

[0170] The fifth encoding module 806 is configured to encode the target speech text corresponding to the target speech segment, obtain the target text features, and call the reference speech synthesis model to generate negative sample speech features based on the target text features.

[0171] The construction module 808 is configured to construct speech synthesis training samples based on reference speech features, target text features, positive sample speech features, and negative sample speech features.

[0172] The above is a schematic scheme of a speech synthesis training sample construction device according to this embodiment. It should be noted that the technical solution of this speech synthesis training sample construction device and the technical solution of the speech synthesis training sample construction method described above belong to the same concept. For details not described in detail in the technical solution of the speech synthesis training sample construction device, please refer to the description of the technical solution of the speech synthesis training sample construction method described above.

[0173] Figure 9 A structural block diagram of a computing device according to one embodiment of this specification is shown. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0174] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 940 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Networks (WLAN) interface, a Wi-MAX (World Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0175] In one embodiment of this specification, the above-described components of the computing device 900 and Figure 9 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 9 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0176] The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 900 can also be a mobile or stationary server.

[0177] The processor 920 is used to execute computer programs / instructions, which, when executed by the processor, implement the steps of the above-mentioned speech synthesis model training method, speech synthesis method, or speech synthesis training sample construction method.

[0178] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the above-mentioned speech synthesis model training method, speech synthesis method, and speech synthesis training sample construction method. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the above-mentioned speech synthesis model training method, speech synthesis method, or speech synthesis training sample construction method.

[0179] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described speech synthesis model training method, speech synthesis method, or speech synthesis training sample construction method.

[0180] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the above-described speech synthesis model training method, speech synthesis method, and speech synthesis training sample construction method. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the above-described speech synthesis model training method, speech synthesis method, or speech synthesis training sample construction method.

[0181] An embodiment of this specification also provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps of the above-described speech synthesis model training method, speech synthesis method, or speech synthesis training sample construction method.

[0182] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the above-mentioned speech synthesis model training method, speech synthesis method, and speech synthesis training sample construction method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the above-mentioned speech synthesis model training method, speech synthesis method, or speech synthesis training sample construction method.

[0183] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0184] Computer instructions include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in computer-readable media can be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0185] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0186] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0187] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A method for training a speech synthesis model, characterized in that, include: Acquire sample speech, wherein the sample speech includes a reference speech segment and a target speech segment; The reference speech segment is encoded to obtain reference speech features, and the target speech segment is encoded to obtain positive sample speech features; The target speech text corresponding to the target speech segment is encoded to obtain target text features, and a reference speech synthesis model is called to generate negative sample speech features based on the target text features; Based on the reference speech features, the target text features, the positive sample speech features, and the negative sample speech features, the parameters of the initial speech synthesis model are adjusted to obtain the trained speech synthesis model.

2. The method according to claim 1, characterized in that, The step of adjusting the parameters of the initial speech synthesis model based on the reference speech features, the target text features, the positive sample speech features, and the negative sample speech features to obtain a trained speech synthesis model includes: Based on the reference speech features and the target text features, conditional input features are generated; Based on the conditional input features, the first positive probability information of the positive sample speech features under the initial speech synthesis model and the first negative probability information of the negative sample speech features under the initial speech synthesis model are determined. Based on the conditional input features, the second positive probability information of the positive sample speech features under the reference speech synthesis model and the second negative probability information of the negative sample speech features under the reference speech synthesis model are determined. Based on the first positive probability information, the first negative probability information, the second positive probability information, and the second negative probability information, determine the preference optimization loss; Based on the preference optimization loss, the parameters of the initial speech synthesis model are adjusted to obtain the trained speech synthesis model.

3. The method according to claim 2, characterized in that, The step of determining the preference optimization loss based on the first positive probability information, the first negative probability information, the second positive probability information, and the second negative probability information includes: By comparing the first positive probability information and the first negative probability information, the first probability difference is determined; By comparing the second positive probability information and the second negative probability information, the second probability difference is determined; The preference optimization loss is determined based on the first probability difference and the second probability difference.

4. The method according to claim 2, characterized in that, The number of negative sample speech features is multiple; The step of determining the preference optimization loss based on the first positive probability information, the first negative probability information, the second positive probability information, and the second negative probability information includes: Based on the first positive probability information, the first negative probability information, the second positive probability information, and the second negative probability information corresponding to each of the negative sample speech features, multiple candidate preference optimization losses are determined respectively, wherein the candidate preference optimization losses correspond one-to-one with the negative sample speech features; The mean of the multiple candidate preference optimization losses is calculated to obtain the preference optimization loss.

5. The method according to claim 2, characterized in that, The positive sample speech features include multiple positive sample speech sub-features arranged in sequence, and the negative sample speech features include multiple negative sample speech sub-features arranged in sequence. The step of determining the first positive probability information of the positive sample speech features and the first negative probability information of the negative sample speech features under the initial speech synthesis model based on the conditional input features includes: Based on the conditional input features, the first prediction probability corresponding to each of the positive sample speech sub-features is determined sequentially according to the arrangement order of the plurality of positive sample speech sub-features, and the first positive probability information is determined based on the plurality of first prediction probabilities. Based on the conditional input features, the second prediction probability corresponding to each negative sample speech sub-feature is determined sequentially according to the arrangement order of the plurality of negative sample speech sub-features, and the first negative probability information is determined based on the plurality of second prediction probabilities.

6. The method according to claim 2, characterized in that, The initial speech synthesis model includes an acoustic synthesis unit and an acoustic decoding unit; The step of adjusting the parameters of the initial speech synthesis model based on the preference optimization loss to obtain the trained speech synthesis model includes: Based on the preference optimization loss, the parameters of the acoustic synthesis unit are adjusted, while the parameters of the acoustic decoding unit remain unchanged, to obtain the trained speech synthesis model.

7. The method according to claim 1, characterized in that, The process of encoding the reference speech segment to obtain reference speech features and encoding the target speech segment to obtain positive sample speech features includes: The same speech encoder is invoked to encode the reference speech segment and the target speech segment respectively, thereby obtaining the reference speech features and the positive sample speech features.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: During the parameter adjustment process of the initial speech synthesis model, the parameter-adjusted initial speech synthesis model is called, and the first verification speech feature is generated based on the verification reference speech features and the verification target text features. The initial speech synthesis model with the adjusted parameters is invoked, and a second verification speech feature is generated based on the verification target text feature; Determine the path similarity between the first verification speech feature and the second verification speech feature; If the path similarity is less than a preset similarity threshold, stop adjusting the parameters of the initial speech synthesis model to obtain the trained speech synthesis model.

9. A speech synthesis method, characterized in that, include: Obtain the target reference audio and the text to be synthesized; The target reference audio is encoded to obtain reference speech features, and the text to be synthesized is encoded to obtain text features. The speech synthesis model is invoked to generate target synthesized speech based on the reference speech features and the text features to be synthesized, wherein the speech synthesis model is trained based on the method described in any one of claims 1 to 8.

10. A method for constructing training samples for speech synthesis, characterized in that, include: Acquire sample speech, wherein the sample speech includes a reference speech segment and a target speech segment; The reference speech segment is encoded to obtain reference speech features, and the target speech segment is encoded to obtain positive sample speech features; The target speech text corresponding to the target speech segment is encoded to obtain target text features, and a reference speech synthesis model is called to generate negative sample speech features based on the target text features; Based on the reference speech features, the target text features, the positive sample speech features, and the negative sample speech features, speech synthesis training samples are constructed.

11. A computing device, characterized in that, include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, It stores a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

13. A computer program product, characterized in that, Includes a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 10.

Images