Speech synthesis method, apparatus, device, and medium

Abstract

The application relates to the technical field of artificial intelligence, is suitable for the question and answer scene of financial technology and medical health, and discloses a speech synthesis method, device, equipment and medium, which comprises the following steps: acquiring an initial text, a reference speech and a speech text corresponding to the reference speech; performing speech length prediction according to the length ratio of the reference speech and the speech text and the length of the initial text to obtain a predicted speech length; performing mask generation according to the predicted speech length to obtain a mask vector sequence; performing feature extraction according to the reference speech to obtain speaker voice features, and performing feature extraction according to the initial text to obtain text semantic features; performing semantic alignment according to the text semantic features and the mask vector sequence to obtain a semantic alignment feature sequence; and performing speech generation according to the semantic alignment feature sequence and the speaker voice features to obtain target speech. The application can improve the stability of alignment between the text and the speech.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applicable to the fields of fintech and healthcare, and particularly to a speech synthesis method, apparatus, device and medium. Background Technology

[0002] Text-to-speech (TTS) is an artificial intelligence technology that converts text into speech output. This technology is a key foundational capability in voice interaction, intelligent customer service, virtual assistants, and multimodal human-computer interaction. For example, in the financial insurance scenario of fintech, TTS technology can be used to build intelligent insurance customer service representatives who can engage in voice dialogues with users to answer their insurance questions. As another example, in the medical insurance scenario of healthcare, TTS technology can be used to generate prompts to remind users to seek medical attention.

[0003] Currently, speech can be synthesized using artificial intelligence models such as non-autoregressive (NAR) models. However, NAR models face difficulties in the alignment modeling stage between text and speech. For example, text sequences and speech sequences are inconsistent in length and temporal structure. Currently, alignment usually relies on explicit manual duration annotation, which can easily lead to stiff and unnatural rhythm in the synthesized speech, meaning that the alignment between text and speech is not very stable. Summary of the Invention

[0004] This application provides a speech synthesis method, apparatus, device, and medium to solve the technical problem of unstable text-speech alignment, aiming to achieve automatic alignment without relying on manual duration annotation, thereby improving the stability of text-speech alignment.

[0005] Firstly, a speech synthesis method is provided, including: Obtain the initial text, the reference speech, and the speech text corresponding to the reference speech; Based on the length ratio of the reference speech to the speech text and the length of the initial text, speech length prediction is performed to obtain the predicted speech length; A mask is generated based on the predicted speech length to obtain a mask vector sequence; wherein the length of the mask vector sequence is the same as the predicted speech length; Feature extraction is performed based on the reference speech to obtain speaker speech features, and feature extraction is performed based on the initial text to obtain text semantic features; Semantic alignment is performed based on the text semantic features and the mask vector sequence to obtain a semantic alignment feature sequence; wherein, the length of the semantic alignment feature sequence is the same as the length of the predicted speech; Speech generation is performed based on the semantic alignment feature sequence and the speaker's speech features to obtain the target speech.

[0006] Secondly, a speech synthesis device is provided, comprising: The data acquisition module is used to acquire the initial text, the reference speech, and the speech text corresponding to the reference speech; The length prediction module is used to predict the speech length based on the length ratio of the reference speech to the speech text and the length of the initial text, so as to obtain the predicted speech length. A mask generation module is used to generate a mask based on the predicted speech length to obtain a mask vector sequence; wherein the length of the mask vector sequence is the same as the predicted speech length; The feature extraction module is used to extract features based on the reference speech to obtain speaker speech features, and to extract features based on the initial text to obtain text semantic features; A semantic alignment module is used to perform semantic alignment based on the text semantic features and the mask vector sequence to obtain a semantic alignment feature sequence; wherein the length of the semantic alignment feature sequence is the same as the length of the predicted speech; The speech generation module is used to generate speech based on the semantic alignment feature sequence and the speaker's speech features to obtain the target speech.

[0007] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.

[0008] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the above-described method.

[0009] In the aforementioned speech synthesis methods, apparatuses, devices, and media, the implicit length ratio between the reference speech and its corresponding speech text can be analyzed by acquiring the initial text, reference speech, and the corresponding speech text. Considering the relatively fixed length ratio between speech and its converted text, the length ratio between the reference speech and its corresponding speech text is consistent with the length ratio between the initial text and the speech synthesized from the initial text. Based on this, the expected length of the speech to be synthesized, i.e., the predicted speech length, can be predicted. Then, a mask vector sequence with the predicted speech length is generated. Semantic alignment is performed based on the text semantic features and the mask vector sequence to obtain a semantic alignment feature sequence with the same length as the predicted speech. This sequence is then combined with the speaker's speech features to generate target speech with speaker characteristics (such as timbre). This allows the mask to constrain the time length of the target speech, automatically aligning the temporal structure of the text and speech without relying on manual time annotation, thereby improving the stability of the alignment between text and speech. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of an application environment for a speech synthesis method according to an embodiment of this application; Figure 2 This is a flowchart illustrating a speech synthesis method in one embodiment of this application; Figure 3 yes Figure 2 A schematic diagram of a specific implementation of step S101; Figure 4 This is another flowchart illustrating the speech synthesis method in one embodiment of this application; Figure 5 yes Figure 4 A schematic diagram of a specific implementation method for step S102; Figure 6 yes Figure 2 A schematic diagram of a specific implementation of step S105; Figure 7 yes Figure 2 A schematic diagram of a specific implementation of step S106; Figure 8 This is another flowchart illustrating the speech synthesis method in one embodiment of this application; Figure 9 This is a schematic diagram of a speech synthesis device in one embodiment of this application; Figure 10 This is a schematic diagram of the structure of a computer device according to one embodiment of this application; Figure 11 This is another structural schematic diagram of a computer device in one embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] The speech synthesis method provided in this application embodiment can be applied to, for example, Figure 1 In this application environment, the client communicates with the server via a network. The server can receive initial text, reference speech, and corresponding speech text from the client, and performs speech synthesis based on the initial text, reference speech, and corresponding speech text to obtain a speech waveform signal, i.e., the target speech, that is semantically consistent with the initial text and matches the speaker features and prosodic characteristics of the reference speech. The server can then feed the target speech back to the client. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The following detailed description of specific embodiments further illustrates this application.

[0014] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the speech synthesis method provided in this application embodiment includes the following steps: S101: Obtain the initial text, reference speech, and the speech text corresponding to the reference speech; S102: Based on the length ratio of the reference speech to the speech text and the length of the initial text, perform speech length prediction to obtain the predicted speech length; S103: Generate a mask based on the predicted speech length to obtain a mask vector sequence; wherein the length of the mask vector sequence is the same as the predicted speech length; S104: Extract features based on the reference speech to obtain speaker speech features, and extract features based on the initial text to obtain text semantic features; S105: Perform semantic alignment based on text semantic features and mask vector sequence to obtain semantic alignment feature sequence; wherein, the length of semantic alignment feature sequence is the same as the length of predicted speech; S106: Generate speech based on semantic alignment feature sequence and speaker speech features to obtain target speech.

[0015] The beneficial effects of this application's embodiments include, but are not limited to: obtaining initial text, reference speech, and the corresponding speech text for the reference speech, so as to subsequently analyze the implicit length ratio between the reference speech and its corresponding speech text. Considering that there is a relatively fixed ratio conversion relationship between the length ratio of speech and its converted text, the length ratio between the reference speech and the corresponding speech text is consistent with the length ratio between the initial text and the speech synthesized based on the initial text. Based on this, the expected length of the speech to be synthesized can be predicted, i.e., the predicted speech length. Then, a mask vector sequence with a length equal to the predicted speech length is generated. Semantic alignment is performed based on the text semantic features and the mask vector sequence to obtain a semantic alignment feature sequence with the same length as the predicted speech. Combined with the speaker's speech features, a target speech with speaker features (such as timbre features) is generated. In this way, the mask can be used to constrain the time length of the target speech. Without relying on manual time annotation or external alignment models, the temporal structure of text and speech can be automatically aligned, improving the stability and generalization ability of the speech synthesis method in multilingual, zero-shot, and complex text scenarios, thereby improving the stability of the alignment between text and speech.

[0016] In step S101 of some embodiments, the initial text refers to the text used for synthesizing speech. Specifically, the initial text can be characters, pinyin, tokens, or other types of text data. For example, the initial text can be "Hello!". It should be noted that the reference speech refers to the speech used to provide speaker features for the speech to be synthesized. The speech text corresponding to the reference speech refers to the text that is semantically consistent with the reference speech. For example, the reference speech can be obtained first, and then converted into text using a speech-to-text conversion model to obtain the speech text. In another embodiment, the speech text can also be obtained first, and then converted into reference speech using a speech-to-text conversion model to obtain the reference speech; this application does not limit this approach.

[0017] In some embodiments, reference speech can be used for zero-shot speaker transfer. It should be noted that zero-shot speaker transfer refers to a technique that synthesizes speech with the timbre of any text from a single reference speech, without requiring training data for the target speaker or model fine-tuning. Zero-shot speaker transfer technology is widely used in speech cloning and cross-speaker TTS (text-to-speech) synthesis.

[0018] In step S102 of some embodiments, the predicted speech length refers to the predicted length of the speech to be synthesized based on the initial text. It should be noted that since the length ratio between the text and the speech generated from it usually remains stable, the length ratio between the reference speech and the text corresponding to the reference speech can be considered the same as the length ratio between the initial text and the target speech. Based on this, the expected length of the target speech, i.e., the predicted speech length N, can be calculated and used as the length of the mask vector sequence.

[0019] In step S103 of some embodiments, the mask vector sequence is a sequence of length N composed of N repeating learnable mask vectors m. It should be noted that the length of the mask vector sequence is the same as the predicted speech length N. In some embodiments, specifically, mask vectors m can be obtained, and N mask vectors m can be concatenated to obtain the mask vector sequence. It should be noted that the mask vector sequence is used to represent the temporal structure of the target speech.

[0020] In step S104 of some embodiments, the speaker speech features are feature vectors obtained by feature extraction from the reference speech. Specifically, the speaker speech features can be obtained by feature extraction from the reference speech using a speech encoder. It should be noted that the text semantic features are feature vectors obtained by feature extraction from the initial text. Specifically, the text semantic features can be obtained by feature extraction from the initial text using a text encoder. In some embodiments, for example, the text encoder can be a Large Language Model (LLM). It should be noted that the LLM includes embedding layers and Transformer layers, and is trained in a supervised manner on large-scale natural language inference datasets and semantic similarity datasets. The LLM learns sentence-level semantic representations through contrastive learning. Its training objective is to maximize the similarity of positive sample pairs (sentences that are semantically similar or inferentially related) and minimize the similarity of negative sample pairs (sentences that are semantically unrelated), that is, semantically similar sentences are close in distance in the vector space, while semantically dissimilar sentences are far apart.

[0021] In step S105 of some embodiments, the semantic alignment feature sequence is a feature vector obtained by jointly modeling the text semantic features and the mask vector sequence. For example, the semantic alignment feature sequence can be obtained by jointly modeling the text semantic features and the mask vector sequence using a Transformer model. In some embodiments, start markers can be introduced before the text semantic features and the mask vector sequence respectively, and then the text semantic features and the mask vector sequence can be jointly modeled using a Transformer model.

[0022] In step S106 of some embodiments, the target speech can be generated by using an artificial intelligence model, such as a stream matching model, to analyze the semantic alignment feature sequence and speaker speech features. It should be noted that the target speech refers to a speech waveform signal that is semantically consistent with the initial text and matches the speaker features (such as timbre, emotion, etc.) and prosodic characteristics of the reference speech.

[0023] In some embodiments, it should be noted that Text-to-Speech (TTS) technology is a key foundational capability in voice interaction, intelligent customer service, virtual assistants, in-vehicle systems, and multimodal human-computer interaction. With the development of deep learning technology, current mainstream TTS systems can be broadly categorized into two types: Autoregressive (AR) models and Non-Autoregressive (NAR) models. Autoregressive TTS models typically predict discrete or continuous representations of speech based on text conditions. Their advantages lie in their strong expressive power, high speech naturalness, and implicit modeling of speech duration and prosodic structure. However, due to their progressively generated structure, autoregressive TTS models inherently suffer from high inference latency and poor parallelism, significantly limiting their performance in real-time synthesis and streaming interaction scenarios. Furthermore, autoregressive models are prone to exposure bias during training and inference and are highly sensitive to the quality of the speech encoder or discretization unit, limiting their stability in multilingual and zero-shot scenarios. To overcome these problems, Non-Autoregressive TTS models have gradually become a research hotspot in recent years. Among them, generation methods based on diffusion models or flow matching can generate speech in a high-quality and parallel manner by modeling the continuous transformation process from simple noise distribution to real speech distribution. These methods are outstanding in zero-sample speech synthesis and speaker transfer. However, current non-autoregressive methods face the following key technical bottlenecks in practical applications: (1) Difficulty in modeling the alignment between text and speech. Text sequences and speech sequences are naturally inconsistent in length and temporal structure. Traditional methods often rely on explicit phoneme duration annotation, forced monotonic alignment, or external duration prediction modules. These methods not only increase the complexity of the system, but also easily lead to stiff rhythm and decreased naturalness of synthesized speech. Some methods simply copy text tokens to match speech length. Although the engineering implementation is simple, there is still the problem of unstable alignment in complex semantics and multilingual scenarios. (2) The generation process based on diffusion or flow matching is computationally intensive. Current flow matching models usually require multi-step iterative calculations in the inference stage. Each step requires the complete execution of conditional encoding and decoding processes, resulting in high inference latency and computational overhead. To reduce the number of inference steps, distillation or teacher-student training mechanisms can be introduced. However, these methods significantly increase training costs and system complexity, hindering engineering implementation and rapid model iteration. Therefore, the embodiments of this application aim to address the problems of unstable text-to-speech alignment and low inference efficiency in text-to-speech synthesis technology without introducing additional alignment annotations and complex distillation processes.

[0024] In some embodiments, the speech synthesis method provided in this application can be applied to intelligent question-and-answer engines such as intelligent customer service or intelligent assistants in various application scenarios. Intelligent question-and-answer engines are typically implemented through a server-side component, which can receive user questions in real time. For example, in the insurance application field, users often ask questions via telephone or online chat, and often need to use an intelligent question-and-answer engine to answer some of their insurance questions in order to improve insurance promotion efficiency and user experience. Specifically, the server-side component can be a server deployed on the insurance company's service website. For example, a user question could be "How to purchase medical insurance," etc. After receiving the user question, the server-side component can further question the user to understand their needs in detail or provide a response, and convert the text of the question or response into speech using speech synthesis technology to generate the target speech.

[0025] In some embodiments, such as Figure 3 As shown, step S101, which involves obtaining the initial text, the reference speech, and the speech text corresponding to the reference speech, includes the following steps: S201: Obtain the initial text, and obtain the initial speech and the speech text corresponding to the initial speech; S202: The initial speech is compressed using a pre-trained target speech compression model to obtain the reference speech.

[0026] The advantage of this embodiment is that by acquiring the initial text, initial speech, and the corresponding speech text, and then performing speech compression processing on the initial speech using a pre-trained target speech compression model, a reference speech is obtained. This converts the original speech into high-information-density, low-bit-rate speech, reduces speech temporal redundancy, reduces the computational burden on the model, and avoids the system complexity introduced by traditional vocoders, thereby improving the efficiency of speech synthesis.

[0027] In step S201 of some embodiments, the initial speech refers to the speech data before compression processing. It should be noted that traditional TTS methods typically use Mel-Frequency Spectrum (MFC) as an intermediate representation for the input speech. This representation has high temporal resolution, significant redundancy, and relies on an additional vocoder, resulting in low speech synthesis efficiency. Therefore, embodiments of this application perform speech compression on the initial speech. It should be noted that Mel-Frequency Spectrum (MFC) is a spectrum that can be used to represent audio information.

[0028] In step S202 of some embodiments, the target speech compression model can be a speech compression model based on a variational autoencoder (VAE). The target speech compression model can be used to map longer speech (such as the initial speech) into shorter, low-frequency latent variable sequences, thereby reducing the bit rate of speech features and providing an efficient representation for subsequent stream matching modeling. In some embodiments, it should be noted that the embodiments of this application use low-bit-rate speech as the generation target, and obtain reference speech through compression, effectively reducing temporal redundancy, reducing the computational burden on the model, and avoiding the system complexity brought by traditional vocoders, which is beneficial to the engineering implementation of end-to-end speech synthesis systems. Furthermore, the embodiments of this application use low-bit-rate speech latent variable representation, reducing speech temporal redundancy and reducing generation complexity from the source.

[0029] In some embodiments, such as Figure 4 As shown, before step S202, the speech synthesis method may further include the following steps: S301: Obtain the second sample speech and the standard compressed speech corresponding to the second sample speech; S302: Calculate the reconstruction loss of the second sample speech using a pre-built initial speech compression model to obtain the reconstruction loss value; S303: The loss regularization term is obtained by calculating the regularization term of the second sample speech and the standard compressed speech through the initial speech compression model; S304: The total compression loss is obtained by weighted summation of the reconstruction loss value and the loss regularization term, and the parameters of the initial speech compression model are adjusted according to the total compression loss value to obtain the target speech compression model.

[0030] The advantage of this embodiment lies in the following: By acquiring a second sample speech and its corresponding standard compressed speech, a reconstruction loss is calculated on the second sample speech using a pre-built initial speech compression model to obtain a reconstruction loss value. A regularization term is then calculated on the second sample speech and the standard compressed speech using the initial speech compression model to obtain a loss regularization term. A weighted sum is then performed based on the reconstruction loss value and the loss regularization term to obtain the total compression loss value. Finally, the parameters of the initial speech compression model are adjusted based on the total compression loss value to obtain the target speech compression model. This allows the trained speech compression model to reduce redundancy in the temporal dimension of speech without losing key features, thereby obtaining low-bitrate speech through speech compression and improving the efficiency of speech synthesis.

[0031] In step S301 of some embodiments, the second sample speech is speech sample data before compression. Standard compressed speech is speech obtained by compressing the second sample speech, and is used as reference data for model training.

[0032] In step S302 of some embodiments, as explained below for step S304, This represents the reconstruction loss value.

[0033] In step S303 of some embodiments, as explained below for step S304, This represents a loss regularization term.

[0034] In step S304 of some embodiments, for example, the total compression loss value is defined as shown in the following formula: , In the formula, This represents the total compression loss value; This represents the second sample speech; This represents the speech output after the initial speech compression model compresses the second sample speech; This indicates the preset weighting parameters; This represents the second sample speech generated by the encoder given the input. Standard compressed speech is generated in time. The probability distribution; This represents standard compressed speech; This represents the prior distribution, usually denoted as the standard normal distribution N(0,1); This represents the positive KL divergence function, which is used to measure the similarity between two probability distributions.

[0035] In some embodiments, such as Figure 5 As shown, in step S102, which involves predicting the speech length based on the length ratio of the reference speech to the speech text and the length of the initial text, the predicted speech length is obtained. This specifically includes the following steps: S401: Obtain the reference speech length of the reference speech and the speech text length of the speech text; S402: Calculate the ratio of the reference speech length to the speech-text length to obtain the speech-text length ratio; S403: Obtain the initial text length of the initial text, and calculate the predicted speech length by multiplying the speech-to-text length ratio and the initial text length; wherein, the ratio of the predicted speech length to the initial text length is consistent with the speech-to-text length ratio.

[0036] The advantage of this embodiment is that by obtaining the reference speech length of the reference speech and the speech text length of the speech text, the ratio is calculated to obtain the speech text length ratio, and then multiplied by the initial text length of the initial text to obtain the predicted speech length. This ensures that the ratio of the predicted speech length to the initial text length is consistent with the speech text length ratio. In this way, the speech length of the initial text can be reasonably estimated based on the actual duration ratio between the reference speech and the speech text, and duration prediction can be achieved without relying on manual duration annotation. Thus, the temporal structure of the text and speech can be automatically aligned without relying on manual duration annotation, thereby improving the stability of the alignment between the text and speech.

[0037] In step S401 of some embodiments, the reference speech length refers to the length of the reference speech, and the speech text length refers to the length of the speech text.

[0038] In step S402 of some embodiments, the speech-to-text length ratio is a ratio calculated with the reference speech length as the numerator and the speech-to-text length as the denominator.

[0039] In step S403 of some embodiments, the ratio of the predicted speech length as the numerator and the initial text length as the denominator is consistent with the speech-to-text length ratio.

[0040] In some embodiments, such as Figure 6 As shown, step S105, which involves semantic alignment based on text semantic features and mask vector sequence to obtain semantic alignment feature sequence, specifically includes the following steps: S501: Obtain the text start marker, and perform feature concatenation based on the text start marker and text semantic features to obtain the marker semantic sequence; S502: Obtain the mask start marker, and perform feature concatenation based on the mask start marker and the mask vector sequence to obtain the marker mask sequence; S503: Perform attention calculation based on the labeled semantic sequence and the labeled mask sequence to obtain the semantic alignment feature sequence; wherein, the length of the semantic alignment feature sequence is the same as the length of the predicted speech.

[0041] The advantage of this embodiment lies in the fact that by concatenating the text start marker with the text semantic features to obtain a marked semantic sequence, and concatenating the mask start marker with the mask vector sequence to obtain a marked mask sequence, an explicit sequence start signal can be introduced. This allows the model to clearly perceive the sequence boundaries during subsequent attention calculations, thereby more stably establishing the correspondence between text semantics and temporal position. Attention calculations are performed based on the marked semantic sequence and the marked mask sequence to obtain a semantic alignment feature sequence, the length of which is the same as the predicted speech length. This allows the attention mechanism to match text semantic features with the speech temporal structure, thereby automatically aligning the temporal structure of text and speech without relying on manual duration annotation, thus improving the stability of the alignment between text and speech.

[0042] In step S501 of some embodiments, the marked semantic sequence is a sequence obtained by concatenating text start markers and text semantic features. In some embodiments, as explained below regarding step S503, This represents a semantic sequence of tags.

[0043] In step S502 of some embodiments, the marker mask sequence is a sequence obtained by feature concatenation based on the mask start marker and the mask vector sequence. In some embodiments, as explained below regarding step S503, This represents a sequence of marker masks.

[0044] In step S503 of some embodiments, attention calculation can be performed on the labeled semantic sequence and the labeled mask sequence using a Transformer model to obtain a semantic alignment feature sequence. It should be noted that the length of the semantic alignment feature sequence is the predicted speech length N, and the semantic alignment feature sequence has implicitly completed the alignment of the text to the speech time axis.

[0045] In some embodiments, specifically, the semantic alignment feature sequence can be calculated using the following formula: , In the formula, Represents a semantically aligned feature sequence; Represents the Transformer model; Indicates the start of text; Representing the semantic features of the text; Indicates the start marker of the mask; It represents N mask vectors The resulting mask vector sequence.

[0046] In some embodiments, such as Figure 7As shown, step S106, which involves generating speech based on the semantic alignment feature sequence and the speaker's speech features to obtain the target speech, specifically includes the following steps: S601: Conditional coding is performed based on the semantic alignment feature sequence, speaker embedding vector, and reference audio to obtain the conditional hidden state; S602: Decode based on the conditional hidden state to obtain the predicted velocity field; S603: The target speech is obtained by performing flow matching speech synthesis on the predicted velocity field through a pre-trained target flow matching model.

[0047] The advantage of this embodiment lies in the fact that conditional encoding is performed based on the semantic alignment feature sequence, speaker embedding vector, and reference audio to obtain a conditional hidden state. This effectively integrates semantic information, speaker identity features, and acoustic details of the reference audio, providing conditional constraints for subsequent speech generation. Decoding is performed based on the conditional hidden state to obtain a predicted velocity field, which transforms abstract semantic and acoustic conditions into specific flow matching path directions. A pre-trained target flow matching model is used to perform flow matching speech synthesis on the predicted velocity field to obtain the target speech. This allows the flow matching model to gradually convert noise into high-quality speech based on the velocity field, automatically aligning the temporal structure of text and speech without relying on manual duration annotation, thereby improving the stability of the alignment between text and speech.

[0048] In step S601 of some embodiments, a conditional encoder can be used to conditionally encode the semantic alignment feature sequence, speaker embedding vector, and reference audio to obtain the conditional hidden state. It should be noted that a conditional encoder is a module in the generative model that incorporates additional conditional information (such as category labels, text, images, etc.) into the encoding process.

[0049] In step S602 of some embodiments, the conditional hidden state can be decoded using a velocity decoder to obtain the predicted velocity field. Specifically, in some embodiments, the loss function of the velocity decoder may include a CTC auxiliary loss function. It should be noted that the velocity decoder is a decoder used to decode and obtain the velocity field, where the velocity field refers to the rate of state change in the generation path from text to speech.

[0050] In step S603 of some embodiments, the target speech can be obtained by performing stream matching speech synthesis based on the predicted velocity field using a pre-trained stream matching model, such as a target stream matching model.

[0051] In some embodiments, such as Figure 8 As shown, before step S603, the speech synthesis method further includes the following steps: S701: Obtain the first sample speech and the sample text corresponding to the first sample speech; S702: Calculate the connection temporal classification loss of the first sample speech and sample text using a pre-built initial stream matching model to obtain the connection temporal classification loss value; S703: The cross-entropy loss is calculated on the first sample speech and sample text using the initial stream matching model to obtain the cross-entropy loss value; S704: The total flow matching loss value is obtained by weighted summation of the cross-entropy loss value and the connection temporal classification loss value, and the parameters of the initial flow matching model are adjusted according to the total flow matching loss value to obtain the target flow matching model.

[0052] The advantage of this embodiment lies in acquiring the first sample speech and the corresponding sample text, and then calculating the connection temporal classification loss of the first sample speech and sample text using a pre-built initial stream matching model to obtain the connection temporal classification loss value. This allows the model to automatically align the temporal relationship between text and speech without relying on manual duration annotation. The initial stream matching model calculates the cross-entropy loss of the first sample speech and sample text to obtain the cross-entropy loss value. The cross-entropy loss value and the connection temporal classification loss value are weighted and summed to obtain the total stream matching loss value. The parameters of the initial stream matching model are then adjusted based on the total stream matching loss value to obtain the target stream matching model. This enables the stream matching model to accurately predict the velocity field and automatically establish the temporal correspondence between text and speech during speech synthesis, thereby automatically aligning the temporal structure of text and speech without relying on manual duration annotation, thus improving the stability of the alignment between text and speech.

[0053] In step S701 of some embodiments, the first sample speech is sample speech used to train the initial stream matching model. The sample text is text that is semantically consistent with the first sample speech. In some embodiments, the first sample speech can be input into a text-to-speech conversion model to generate sample text. The first sample speech and sample text can also be obtained in other ways, and are not limited thereto.

[0054] In step S702 of some embodiments, it should be noted that the Connectionist Temporal Classification (CTC) loss value, also known as the connectionist temporal classification loss value, is a loss value used to enable the model to learn reasonable temporal alignment.

[0055] In step S703 of some embodiments, the cross-entropy loss value can be obtained by calculating the cross-entropy loss of the first sample speech and sample text based on the cross-entropy loss function using the initial stream matching model. It should be noted that the cross-entropy loss function is a loss function used to measure the difference between the model's predicted probability distribution and the true probability distribution.

[0056] In step S704 of some embodiments, the total flow matching loss is a weighted sum of the cross-entropy loss and the connection temporal classification loss. The target flow matching model is a model obtained by adjusting the parameters of the initial flow matching model.

[0057] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0058] In one embodiment, a speech synthesis device is provided, which corresponds one-to-one with the speech synthesis methods described in the above embodiments. For example... Figure 9 As shown, the speech synthesis device includes a data acquisition module 801, a length prediction module 802, a mask generation module 803, a feature extraction module 804, a semantic alignment module 805, and a speech generation module 806. Detailed descriptions of each functional module are as follows: Data acquisition module 801 is used to acquire initial text, reference speech, and speech text corresponding to the reference speech; The length prediction module 802 is used to predict the speech length based on the length ratio of the reference speech to the speech text and the length of the initial text, so as to obtain the predicted speech length. The mask generation module 803 is used to generate a mask based on the predicted speech length to obtain a mask vector sequence; wherein the length of the mask vector sequence is the same as the predicted speech length. The feature extraction module 804 is used to extract features based on the reference speech to obtain speaker speech features, and to extract features based on the initial text to obtain text semantic features; The semantic alignment module 805 is used to perform semantic alignment based on the text semantic features and the mask vector sequence to obtain a semantic alignment feature sequence; wherein the length of the semantic alignment feature sequence is the same as the length of the predicted speech. The speech generation module 806 is used to generate speech based on the semantic alignment feature sequence and the speaker's speech features to obtain the target speech.

[0059] This application provides a speech synthesis apparatus that acquires initial text, reference speech, and the corresponding speech text to facilitate subsequent analysis of the implicit length ratio between the reference speech and its corresponding speech text. Considering the relatively fixed length ratio between speech and its converted text, the length ratio between the reference speech and its corresponding speech text is consistent with the length ratio between the initial text and the speech synthesized from the initial text. Based on this, the expected length of the speech to be synthesized, i.e., the predicted speech length, can be predicted. Then, a mask vector sequence with the predicted speech length is generated. Semantic alignment is performed based on the text semantic features and the mask vector sequence to obtain a semantic alignment feature sequence with the same length as the predicted speech. This sequence is then combined with speaker speech features to generate target speech with speaker characteristics (such as timbre). This allows the mask to constrain the temporal length of the target speech, automatically aligning the temporal structure of the text and speech without relying on manual duration annotation, thereby improving the stability of the alignment between text and speech.

[0060] Specific limitations regarding the speech synthesis device can be found in the limitations of the intelligent question-answering method described above, and will not be repeated here. Each module in the aforementioned speech synthesis device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0061] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 10 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a speech synthesis method on the server side.

[0062] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 11As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a speech synthesis method on the client side.

[0063] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Obtain the initial text, the reference speech, and the speech text corresponding to the reference speech; Based on the length ratio of the reference speech to the speech text and the length of the initial text, the speech length is predicted to obtain the predicted speech length. A mask is generated based on the predicted speech length, resulting in a mask vector sequence; the length of the mask vector sequence is the same as the predicted speech length. Feature extraction is performed based on the reference speech to obtain the speaker's speech features, and feature extraction is performed based on the initial text to obtain the text semantic features; Semantic alignment is performed based on text semantic features and mask vector sequences to obtain a semantic alignment feature sequence; the length of the semantic alignment feature sequence is the same as the length of the predicted speech. Speech generation is performed based on semantic alignment feature sequences and speaker speech features to obtain the target speech.

[0064] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Obtain the initial text, the reference speech, and the speech text corresponding to the reference speech; Based on the length ratio of the reference speech to the speech text and the length of the initial text, the speech length is predicted to obtain the predicted speech length. A mask is generated based on the predicted speech length, resulting in a mask vector sequence; the length of the mask vector sequence is the same as the predicted speech length. Feature extraction is performed based on the reference speech to obtain the speaker's speech features, and feature extraction is performed based on the initial text to obtain the text semantic features; Semantic alignment is performed based on text semantic features and mask vector sequences to obtain a semantic alignment feature sequence; the length of the semantic alignment feature sequence is the same as the length of the predicted speech. Speech generation is performed based on semantic alignment feature sequences and speaker speech features to obtain the target speech.

[0065] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0066] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0067] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0068] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0069] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A speech synthesis method, characterized in that, include: Obtain the initial text, the reference speech, and the speech text corresponding to the reference speech; Based on the length ratio of the reference speech to the speech text and the length of the initial text, speech length prediction is performed to obtain the predicted speech length; A mask is generated based on the predicted speech length to obtain a mask vector sequence; wherein the length of the mask vector sequence is the same as the predicted speech length; Feature extraction is performed based on the reference speech to obtain speaker speech features, and feature extraction is performed based on the initial text to obtain text semantic features; Semantic alignment is performed based on the text semantic features and the mask vector sequence to obtain a semantic alignment feature sequence; wherein, the length of the semantic alignment feature sequence is the same as the length of the predicted speech; Speech generation is performed based on the semantic alignment feature sequence and the speaker's speech features to obtain the target speech.

2. The method according to claim 1, characterized in that, The step of predicting the predicted speech length based on the length ratio of the reference speech to the speech text and the length of the initial text includes: Obtain the reference speech length of the reference speech, and obtain the speech text length of the speech text; The speech-to-text length ratio is calculated by comparing the reference speech length with the speech-to-text length. The initial text length of the initial text is obtained, and the predicted speech length is calculated by multiplying the speech-to-text length ratio and the initial text length; wherein the ratio of the predicted speech length to the initial text length is consistent with the speech-to-text length ratio.

3. The method according to claim 1, characterized in that, The step of generating speech based on the semantic alignment feature sequence and the speaker's speech features to obtain the target speech includes: Conditional encoding is performed based on the semantic alignment feature sequence, speaker embedding vector, and reference audio to obtain the conditional hiding state; Decode the predicted velocity field based on the conditional hidden state; The predicted velocity field is subjected to flow matching speech synthesis using a pre-trained target flow matching model to obtain the target speech.

4. The method according to claim 3, characterized in that, Before performing stream-matched speech synthesis on the predicted velocity field using a pre-trained target stream matching model to obtain the target speech, the method further includes: Obtain the first sample speech and the sample text corresponding to the first sample speech; The connection temporal classification loss is calculated by using a pre-built initial stream matching model to obtain the connection temporal classification loss value; The cross-entropy loss value is obtained by calculating the cross-entropy loss between the first sample speech and the sample text using the initial stream matching model. The total flow matching loss value is obtained by weighted summation of the cross-entropy loss value and the connection temporal classification loss value, and the parameters of the initial flow matching model are adjusted according to the total flow matching loss value to obtain the target flow matching model.

5. The method according to any one of claims 1 to 4, characterized in that, The step of performing semantic alignment based on the text semantic features and the mask vector sequence to obtain a semantic alignment feature sequence includes: Obtain the text start marker, and perform feature concatenation based on the text start marker and the text semantic features to obtain the marker semantic sequence; Obtain the mask start marker, and perform feature concatenation based on the mask start marker and the mask vector sequence to obtain the marker mask sequence; Attention is calculated based on the labeled semantic sequence and the labeled mask sequence to obtain the semantic alignment feature sequence; wherein the length of the semantic alignment feature sequence is the same as the length of the predicted speech.

6. The method according to any one of claims 1 to 4, characterized in that, The process of obtaining the initial text, the reference speech, and the speech text corresponding to the reference speech includes: Obtain the initial text, and obtain the initial speech and the speech text corresponding to the initial speech; The initial speech is compressed using a pre-trained target speech compression model to obtain the reference speech.

7. The method as described in claim 6, characterized in that, Before performing speech compression processing on the initial speech using a pre-trained target speech compression model to obtain the reference speech, the method further includes: Obtain the second sample speech and the standard compressed speech corresponding to the second sample speech; The reconstruction loss value is obtained by calculating the reconstruction loss of the second sample speech using a pre-constructed initial speech compression model. The loss regularization term is obtained by performing regularization on the second sample speech and the standard compressed speech using the initial speech compression model. The total compression loss value is obtained by weighted summation of the reconstruction loss value and the loss regularization term, and the parameters of the initial speech compression model are adjusted according to the total compression loss value to obtain the target speech compression model.

8. A speech synthesis device, characterized in that, include: The data acquisition module is used to acquire the initial text, the reference speech, and the speech text corresponding to the reference speech; The length prediction module is used to predict the speech length based on the length ratio of the reference speech to the speech text and the length of the initial text, so as to obtain the predicted speech length. A mask generation module is used to generate a mask based on the predicted speech length to obtain a mask vector sequence; wherein the length of the mask vector sequence is the same as the predicted speech length; The feature extraction module is used to extract features based on the reference speech to obtain speaker speech features, and to extract features based on the initial text to obtain text semantic features; A semantic alignment module is used to perform semantic alignment based on the text semantic features and the mask vector sequence to obtain a semantic alignment feature sequence; wherein the length of the semantic alignment feature sequence is the same as the length of the predicted speech; The speech generation module is used to generate speech based on the semantic alignment feature sequence and the speaker's speech features to obtain the target speech.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images