Text-to-speech synthesis method and device, equipment and medium

Abstract

The invention belongs to the field of artificial intelligence, and relates to a text-to-speech synthesis method and device, equipment and a medium, and the method comprises the steps: obtaining a normalized text of a to-be-synthesized speech and a reference speech of a target speaker, and coding the reference speech to generate a target speaker vector; text hidden representation and Gaussian distribution parameters of all text units of the normalized text are obtained, time length randomization noise is obtained through sampling according to the length of a text hidden representation sequence, and a generator is used for predicting a time length sequence. The method comprises the following steps: extracting a text unit, sampling according to the length of the text unit to obtain aligned randomized noise, determining an alignment relationship between the text unit and a potential speech segment through monotonous alignment search with noise, and finally generating a synthetic speech according with the characteristics of a target speaker by using a preset normalized stream model based on a text hidden representation sequence and the like. The method can be applied to the business fields of financial science and technology, insurance, medical treatment and the like, and the speech synthesis quality can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and is applied to online processing business scenarios such as fintech, insurance, and healthcare. In particular, it relates to a text-to-speech synthesis method, apparatus, device, and medium. Background Technology

[0002] In recent years, the rapid development of deep neural network technology has driven significant progress in text-to-speech technology, bringing groundbreaking changes to the field of speech synthesis. Traditional speech synthesis systems often employ a two-stage pipeline architecture. This architecture first generates manually defined intermediate speech representations, such as Mel spectrograms, based on the input text, and then converts these intermediate representations into the original speech waveform using a vocoder. Although this two-stage architecture simplifies the model training process, it has significant technical drawbacks in practical applications: the manually defined nature of intermediate features makes them difficult to become the optimal representation learned internally by the model, and the prediction errors in the first stage accumulate and propagate in the second stage, directly leading to a decrease in the quality of the final synthesized speech. Furthermore, the generation of intermediate features adds additional computational overhead, reducing system efficiency. Especially in financial scenarios, customer service representatives need to broadcast key information such as policy terms, claims progress, and account balances to users via voice. The speech stuttering and semantic ambiguity caused by the two-stage architecture can easily lead users to question the accuracy of the information and even increase the risk of financial disputes.

[0003] To overcome the inherent limitations of two-stage systems, single-stage speech synthesis models have emerged, aiming to achieve a direct mapping from text to speech waveforms and simplify the synthesis process. However, existing single-stage models still face several technical bottlenecks: the stream-based random duration predictor relied upon in the duration prediction stage suffers from high computational complexity and a cumbersome training process, affecting the overall operational efficiency of the model; during alignment and duration modeling, "whitespace" is typically introduced to expand the input text, increasing the complexity of input processing; simultaneously, existing models generally rely heavily on the conversion from text to phoneme sequences, hindering the realization of a fully end-to-end single-stage synthesis goal. In financial scenarios, this manifests as the model's inability to quickly respond to the real-time voice demands of massive numbers of users, such as during peak periods of automated customer service, and the speech redundancy caused by "whitespace" prolongs the broadcast time of policy notifications, risk warnings, and other content, reducing the efficiency of financial services. Summary of the Invention

[0004] The purpose of this application is to propose a text-to-speech synthesis method, apparatus, computer device, and storage medium to solve the problems of complex single-stage model duration prediction and difficulty in achieving complete end-to-end synthesis in current synthesized speech technology.

[0005] Firstly, a text-to-speech synthesis method is provided, which adopts the following technical solution: The process involves: acquiring normalized text and reference speech of the target speaker; performing speech feature encoding on the reference speech to generate a target speaker vector; obtaining the text hidden representation and Gaussian distribution parameters for each text unit in the normalized text based on the normalized text and the target speaker vector; sampling from a standard normal distribution based on the sequence length of the text hidden representation sequence to obtain duration randomization noise, with the text hidden representation sequence consisting of all text hidden representations; using an adversarial learning-based generator to predict duration based on the text hidden representation sequence and duration randomization noise to obtain a predicted duration sequence corresponding to the order of each text unit in the normalized text; sampling from a standard normal distribution based on the sequence length of the predicted duration sequence to obtain alignment randomization noise; obtaining the alignment relationship between text units and potential speech segments through noisy monotonic alignment search based on the Gaussian distribution parameters, the target speaker vector, the predicted duration sequence, and the alignment randomization noise; and generating synthesized speech that conforms to the characteristics of the target speaker using a pre-defined normalized flow model based on the text hidden representation sequence, the alignment relationship, and the target speaker vector.

[0006] Secondly, a text-to-speech synthesis device is provided, which adopts the following technical solution: The first acquisition module is used to acquire the normalized text of the speech to be synthesized and the reference speech of the target speaker; The first generation module is used to perform speech feature encoding processing on the reference speech, generate the target speaker vector, and obtain the text hidden representation and Gaussian distribution parameters of each text unit of the normalized text based on the normalized text and the target speaker vector. The first sampling module is used to sample from a standard normal distribution based on the sequence length of the text hidden representation sequence to obtain duration randomized noise. The text hidden representation sequence is composed of all text hidden representations. The prediction module is used to predict the duration based on the text hidden representation sequence and duration randomization noise, using an adversarial learning-based generator to obtain the predicted duration sequence corresponding to the order of each text unit in the normalized text. The second sampling module is used to sample from the standard normal distribution based on the sequence length of the predicted duration sequence to obtain aligned randomized noise; The alignment module is used to obtain the alignment relationship between text units and potential speech segments through noisy monotonic alignment search based on Gaussian distribution parameters, target speaker vector, prediction duration sequence and alignment randomization noise. The second generation module is used to generate synthesized speech that conforms to the characteristics of the target speaker based on the text hidden representation sequence, alignment relationship, and target speaker vector, using a preset normalized flow model.

[0007] Thirdly, a computer device is provided, which adopts the following technical solution: The process involves: acquiring normalized text and reference speech of the target speaker; performing speech feature encoding on the reference speech to generate a target speaker vector; obtaining the text hidden representation and Gaussian distribution parameters for each text unit in the normalized text based on the normalized text and the target speaker vector; sampling from a standard normal distribution based on the sequence length of the text hidden representation sequence to obtain duration randomization noise, with the text hidden representation sequence consisting of all text hidden representations; using an adversarial learning-based generator to predict duration based on the text hidden representation sequence and duration randomization noise to obtain a predicted duration sequence corresponding to the order of each text unit in the normalized text; sampling from a standard normal distribution based on the sequence length of the predicted duration sequence to obtain alignment randomization noise; obtaining the alignment relationship between text units and potential speech segments through noisy monotonic alignment search based on the Gaussian distribution parameters, the target speaker vector, the predicted duration sequence, and the alignment randomization noise; and generating synthesized speech that conforms to the characteristics of the target speaker using a pre-defined normalized flow model based on the text hidden representation sequence, the alignment relationship, and the target speaker vector.

[0008] Fourthly, a computer-readable storage medium is provided, which adopts the following technical solution: The process involves: acquiring normalized text and reference speech of the target speaker; performing speech feature encoding on the reference speech to generate a target speaker vector; obtaining the text hidden representation and Gaussian distribution parameters for each text unit in the normalized text based on the normalized text and the target speaker vector; sampling from a standard normal distribution based on the sequence length of the text hidden representation sequence to obtain duration randomization noise, with the text hidden representation sequence consisting of all text hidden representations; using an adversarial learning-based generator to predict duration based on the text hidden representation sequence and duration randomization noise to obtain a predicted duration sequence corresponding to the order of each text unit in the normalized text; sampling from a standard normal distribution based on the sequence length of the predicted duration sequence to obtain alignment randomization noise; obtaining the alignment relationship between text units and potential speech segments through noisy monotonic alignment search based on the Gaussian distribution parameters, the target speaker vector, the predicted duration sequence, and the alignment randomization noise; and generating synthesized speech that conforms to the characteristics of the target speaker using a pre-defined normalized flow model based on the text hidden representation sequence, the alignment relationship, and the target speaker vector.

[0009] Compared with existing technologies, the embodiments of this application have the following main advantages: By combining an adversarial learning generator with duration randomization noise for duration prediction, replacing the traditional stream-based random duration predictor, the computational complexity and training difficulty of duration prediction are reduced, and the efficiency of duration prediction is improved. Simultaneously, using normalized text as input, there is no need to convert the text into a phoneme sequence or introduce "whitespace" to expand the input. Accurate alignment of text units and potential speech segments is achieved through noisy monotonic alignment search. Combined with a normalized flow model, the target speech is directly generated, realizing complete end-to-end synthesis from normalized text to speech waveforms. This effectively avoids error accumulation and additional computational overhead caused by intermediate features, ensuring the quality of synthesized speech. Attached Figure Description

[0010] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2 A flowchart of an embodiment of the text-to-speech synthesis method according to this application; Figure 3 This is a schematic diagram of one embodiment of the text-to-speech synthesis apparatus according to this application; Figure 4 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0012] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0013] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0014] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0015] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers.

[0016] Server 103 can be a server that provides various services, such as a backend server that supports the pages displayed on terminal device 101.

[0017] It should be noted that the text-to-speech synthesis method provided in this application embodiment is generally executed by a server / terminal device, and correspondingly, the text-to-speech synthesis device is generally located in the server / terminal device.

[0018] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0019] Continue to refer to Figure 2 A flowchart of an embodiment of the text-to-speech synthesis method according to this application is shown. The text-to-speech synthesis method includes the following steps: Step S201: Obtain the normalized text of the speech to be synthesized and the reference speech of the target speaker.

[0020] Normalized text is the text form after the original text to be synthesized has been standardized in terms of format. The process may include operations such as removing redundant symbols and unifying formats to eliminate the format differences of the original text in order to adapt to the subsequent text unit splitting and encoding process.

[0021] In this context, the target speaker refers to the specific subject whose speech characteristics (including timbre, rhythm, and pronunciation habits) need to be simulated in the speech synthesis task; it is the style target of the synthesized speech. The reference speech is the speech sample data of the target speaker, typically one or more valid speech recordings of the target speaker. Its core purpose is to provide a data foundation for extracting the target speaker's personalized features. For example, a 10-second audio recording of the target speaker "User A" reading aloud serves as the reference speech for this speech synthesis task.

[0022] Step S202: Perform speech feature encoding on the reference speech to generate the target speaker vector. Based on the normalized text and the target speaker vector, obtain the text hiding representation and Gaussian distribution parameters of each text unit in the normalized text.

[0023] Speech feature encoding is a technical process for extracting and vectorizing features from reference speech. It can be implemented using a pre-trained audio encoder. Its core is to convert continuous speech signals into a numerical vector sequence that can represent the speaker's features. The target speaker vector is a low-dimensional numerical vector extracted from the reference speech and generated by speech feature encoding. Its core function is to represent the target speaker's personalized speech features (such as unique timbre and prosodic habits). This feature is then injected into subsequent processes such as text encoding and speech generation to achieve personalized speech synthesis.

[0024] The text hiding representation is a high-dimensional feature vector obtained after the text unit is processed by a speaker-conditionally conditioned text encoder. It not only contains the semantic information of the text unit but also incorporates the personalized features of the target speaker vector. The Gaussian distribution parameters are the statistical parameters of the Gaussian distribution that the speech features corresponding to the text unit follow. They can include the mean and variance and are output by the speaker-conditionally conditioned text encoder. Their core purpose is to describe the probability distribution of the speech latent variables corresponding to the text unit.

[0025] Step S203: Based on the sequence length of the text hiding representation sequence, sample from the standard normal distribution to obtain duration randomized noise. The text hiding representation sequence is composed of all text hiding representations.

[0026] The text hiding representation sequence is a vector sequence formed by arranging the text hiding representations corresponding to all text units after normalized text is split into text units in the original order of the text units.

[0027] Among them, the standard normal distribution is a continuous probability distribution with a mean of 0 and a variance of 1. Its probability density function is symmetric and bell-shaped. Its core use is to provide a probability distribution basis for generating random noise, ensuring the randomness and distribution rationality of the noise. Sampling is the process of selecting random values from a specified probability distribution (such as the standard normal distribution). Its core use is to generate random input parameters, such as duration randomization noise and alignment randomization noise, to increase the diversity and robustness of the model output. Duration randomization noise is a sequence of random values sampled from the standard normal distribution, and its length is consistent with the text hidden representation sequence. Its core use is to be injected into the duration prediction process, increasing the diversity of duration prediction results and avoiding the problem of rigid prosody caused by overly single prediction results.

[0028] Step S204: Based on the text hidden representation sequence and the duration randomization noise, use a generator based on adversarial learning to perform duration prediction, and obtain a predicted duration sequence corresponding to the order of each text unit of the normalized text.

[0029] Among them, the generator is a neural network module constructed based on the adversarial learning framework. Its core function is to receive input features, such as the text hidden representation sequence and the duration randomization noise, and output prediction results, such as the predicted duration sequence. Through adversarial training, the accuracy and diversity of the prediction results are improved. Duration prediction is a technical process of predicting the corresponding duration of each text unit in the normalized text in the synthesized speech, which is the core link of speech prosody modeling. Its prediction results directly affect the prosody naturalness of the synthesized speech. For example, predicting that the speech duration corresponding to "今" in "今天天气好" is 0.2 seconds and the duration corresponding to "天" is 0.15 seconds is the result of duration prediction.

[0030] Among them, the predicted duration sequence is a numerical sequence formed by arranging the predicted durations corresponding to each text unit in the original order of the text units, and its length is consistent with the number of text units. Its core use is to constrain the length of the subsequent target speech latent variable sequence and provide a duration basis for the alignment of the text and the speech latent segments. For example, the numerical sequence formed by arranging "今(0.2s)、天(0.15s)、天(0.15s)、气(0.1s)、好(0.2s)" in order is the predicted duration sequence corresponding to this normalized text.

[0031] Step S205: Based on the sequence length of the predicted duration sequence, sample from the standard normal distribution to obtain alignment randomization noise.

[0032] Among them, the alignment randomization noise is a sequence of random values sampled from the standard normal distribution, and its length is consistent with the predicted duration sequence. Its core use is to be injected into the alignment search process, increasing the flexibility of alignment path exploration, avoiding falling into local optimal alignment results, and improving the rationality of the alignment relationship.

[0033] Step S206: Based on Gaussian distribution parameters, target speaker vector, predicted duration sequence and alignment randomization noise, the alignment relationship between text units and potential speech segments is obtained through noisy monotonic alignment search.

[0034] Among them, the noisy monotonic alignment search is an alignment method that introduces random noise into the alignment search process while maintaining the consistency of the order of text units and potential speech segments. The core process includes steps such as determining the segment range, combining and calculating the matching degree, and recursively accumulating the matching degree with noise, and finally obtaining the optimal mapping relationship between text and speech.

[0035] The alignment relationship refers to the one-to-one mapping relationship between text units of normalized text and latent variable segments of target speech. It is determined by noisy monotonic alignment search. Its core purpose is to establish the association between text semantics and speech features, providing an accurate text-speech mapping basis for subsequent speech generation.

[0036] Step S207: Based on the text hiding representation sequence, alignment relationship, and target speaker vector, a pre-defined normalized flow model is used to generate synthesized speech that conforms to the characteristics of the target speaker.

[0037] Among them, the normalized flow model is a probabilistic generation model based on reversible transformation. Its core function is to transform simple distributions (such as the standard normal distribution) into complex distributions (such as speech feature distributions). It is used to generate speech waveforms that conform to the characteristics of the target speaker based on the text hidden representation sequence, alignment relationship and target speaker vector, so as to realize end-to-end speech synthesis.

[0038] The synthesized speech is the final output of this scheme. It is audio data generated by a normalized flow model based on normalized text, target speaker vector and alignment relationship. Its core feature is to match the speech style of the target speaker (such as timbre and prosody) while accurately conveying the semantic information of the normalized text.

[0039] This application's embodiments utilize an adversarial learning generator combined with duration-randomized noise for duration prediction, replacing traditional stream-based random duration predictors. This reduces the computational complexity and training difficulty of duration prediction, while improving its efficiency. Furthermore, by using normalized text as input, it eliminates the need to convert the text into a phoneme sequence or introduce whitespace to expand the input. Precise alignment between text units and potential speech segments is achieved through noisy monotonic alignment search. Combined with a normalized stream model, the target speech is directly generated, realizing complete end-to-end synthesis from normalized text to speech waveforms. This effectively avoids error accumulation and additional computational overhead caused by intermediate features, ensuring the quality of the synthesized speech.

[0040] In some optional implementations of this embodiment, step 202, based on the normalized text and the target speaker vector, obtains the text hiding representation and Gaussian distribution parameters of each text unit of the normalized text, specifically including the following steps: The normalized text is divided into multiple text units according to a preset splitting rule. The split text units are then arranged into a text unit sequence according to the text order of the normalized text. The text unit sequence is then fused with the target speaker vector to obtain a text input with fused speaker features. A speaker-conditionally adjusted text encoder is used to encode and map the text input, generating a text hidden representation corresponding to each text unit of the normalized text, as well as the Gaussian distribution parameters of the speech features corresponding to each text unit.

[0041] The splitting rules are the pre-defined logic for splitting normalized text into units. These rules can be set based on the semantic boundaries of the text, such as the granularity requirements of Chinese characters, words, symbols, or speech synthesis, ensuring that the split units possess independent semantics while also being suitable for the granularity of subsequent encoding and duration prediction. The text unit sequence is a set of units formed by arranging the normalized text according to the original text's reading order after processing by the splitting rules.

[0042] The text input is the data obtained by fusing the text unit sequence and the target speaker vector through feature fusion. It contains both the semantic information of the text units and the personalized features of the target speaker. The speaker-conditional text encoder is a text encoding model that embeds a speaker feature adjustment mechanism. During the encoding process, the adjustment effect of the target speaker vector is introduced, such as through attention layer injection and feedforward layer concatenation, so that the output text representation carries both semantic information and speaker features. The encoding mapping is the feature transformation process of the speaker-conditional text encoder on the input data. Specifically, it maps the text input (discrete unit or vector form) with fused speaker features into a high-dimensional continuous text hidden representation and performs calculations on Gaussian distribution parameters.

[0043] In one example, the following takes the voice of the intelligent customer service "Specialist C" in the field of synthetic financial insurance, "Your insurance claim application has been accepted", as an example to elaborate on the specific implementation process. First, obtain the text to be synthesized, "Your insurance claim application has been accepted". After preprocessing to obtain the normalized text, use the "Chinese single-character splitting rule" to split it into a sequence of text units: ["您", "的", "保", "险", "理", "赔", "申", "请", "已", "受", "理"], a total of 11 text units, arranged in the original text order to preserve the semantic logic. Subsequently, obtain the 5-second service voice of the intelligent customer service "Specialist C" as the reference voice, and obtain the target speaker vector with a dimension of 256 through encoding. Concatenate the above text unit sequence and the vector through a fully connected layer to achieve feature fusion, and obtain a text input that combines speaker features (each text unit corresponds to a fusion vector with a dimension of 512 + 256). Finally, use a text encoder with speaker-conditioned adjustment based on the improved Transformer to encode and map this text input: output the 512-dimensional text hidden representation corresponding to each text unit (such as the hidden representation vector corresponding to "理"), and at the same time output the Gaussian distribution parameters of the voice features corresponding to each unit, such as the mean μ = 0.55 and variance σ = 0.13 corresponding to "赔".

[0044] In the embodiment of this application, the normalized text is split into an ordered sequence of text units through the splitting rule, which not only ensures that the granularity of the text semantics adapts to subsequent processing but also maintains the original temporal logic of the text. Feature fusion is performed on this sequence and the target speaker vector, enabling the text input to carry both semantic information and speaker personalization features, avoiding the separation of semantic and style features. Then, the fused input is encoded and mapped through a text encoder with speaker-conditioned adjustment. The generated text hidden representation not only accurately conveys the text semantics but also incorporates the speaker features. At the same time, the output Gaussian distribution parameters provide a probability basis for subsequent voice segment matching. This process does not rely on text-to-phoneme conversion, simplifies the input processing link, and also lays a semantic carrier foundation with speaker features for subsequent fully end-to-end synthesis.

[0045] In some optional implementation manners, before step 204, based on the text hidden representation sequence and duration randomized noise, using a generator based on adversarial learning to predict the duration and obtaining a predicted duration sequence corresponding to the order of each text unit of the normalized text, the following steps are further specifically included: Acquire the text and speech data to be trained. Based on the text and speech data, obtain text hidden representation sequence training data, real duration sequence training data, and duration randomized noise training data. Input the text hidden representation sequence training data and duration randomized noise training data into a preset generator to obtain predicted duration sequence training data. Input the text hidden representation sequence training data and real duration sequence training data into a preset discriminator to obtain a first discrimination result. Input the text hidden representation sequence training data and predicted duration sequence training data into the discriminator to obtain a second discrimination result. Based on the first discrimination result and the second... The discriminator loss and generator loss are calculated based on the discrimination results. It is then determined whether the discriminator loss and generator loss meet the preset convergence conditions. If not, the parameters of the discriminator and generator are updated by backpropagation based on the discriminator loss and generator loss, respectively. New training text and speech data are acquired, and the steps of generating new predicted duration sequence training data through the generator for discrimination and discriminating new real duration sequence training data through the discriminator are repeated until the discriminator loss and generator loss of the new round meet the convergence conditions. The generator corresponding to the convergence condition is determined as the generator based on adversarial learning.

[0046] The training data consists of text and speech data, which are sample datasets used to train the generator and discriminator in the adversarial learning framework. These datasets are composed of pairs of text data and corresponding speech data, providing associated samples of text semantics and speech features to ensure the trained model can accurately predict the speech duration corresponding to each text unit. The text hidden representation sequence training data is used as one of the input features for the generator and discriminator, providing a high-dimensional representation of the text semantics. The real duration sequence training data is the actual speech duration sequence corresponding to each text unit extracted from the training speech data. It serves as the real label for model training, used to calculate the discriminator loss and generator loss, and to supervise model learning. The duration randomized noise training data is a random numerical sequence sampled from a standard normal distribution based on the length of the text hidden representation sequence training data. Its core purpose is to inject into the generator's training process, improving the diversity and robustness of the generator's predicted duration.

[0047] The pre-defined generator is a neural network module within the adversarial learning framework used to predict duration sequences. Initially built based on a pre-defined architecture, its core function is to receive training data of text hidden representation sequences and duration-randomized noise training data, and output predicted duration sequence training data. The purpose of this predicted duration sequence training data is to compare it with the actual duration sequence training data, providing a basis for calculating the model's loss. For example, given the training data related to "Your policy is in effect," the generator outputs duration sequences of "0.18s, 0.12s, 0.14s, 0.16s, 0.19s," which are the predicted duration sequence training data.

[0048] The first discrimination result is the discriminator's judgment on whether the input is a real duration sequence after receiving training data of text hidden representation sequences and training data of real duration sequences. It can be presented in the form of probability values. Its core purpose is to participate in the calculation of the discriminator's loss, reflecting the discriminator's ability to recognize real data. The second discrimination result is also the discriminator's judgment on whether the input is a real duration sequence after receiving training data of text hidden representation sequences and training data of predicted duration sequences. It is also presented in the form of probability values. Its core purpose is to participate in the calculation of discriminator loss and generator loss, reflecting the discriminator's ability to distinguish generated data.

[0049] The discriminator loss is a quantitative metric calculated based on the first and second discrimination results. It measures the discriminator's ability to distinguish between the real and predicted duration sequences, and its core purpose is to serve as the basis for updating the discriminator parameters. The generator loss is a quantitative metric calculated based on the second discrimination result. It measures the closeness between the predicted duration sequence output by the generator and the real duration sequence, and its core purpose is to serve as the basis for updating the generator parameters. The convergence condition is a preset criterion for determining whether model training is complete. It is based on the numerical thresholds of the discriminator and generator losses, the trend of loss value changes, etc. Its core purpose is to terminate the training process and determine the final adversarial learning-based generator.

[0050] In one example, using the scenario of a medical insurance reimbursement progress notification, the training process of a generator based on adversarial learning is illustrated. First, text and speech data to be trained are acquired. The text is "Your medical insurance reimbursement application has been approved, and the reimbursement amount will arrive in 3 business days," and the corresponding speech is a notification voice recorded by a professional customer service representative. Based on this data, a text encoder with speaker condition adjustment is used to obtain text hidden representation sequence training data. The actual duration of each text unit is extracted from the speech, resulting in a real duration sequence training data of "0.2s, 0.1s, 0.15s, 0.15s, 0.2s, 0.18s, 0.2s, 0.15s, 0.1s, 0.2s". Randomized noise training data is obtained by sampling from a standard normal distribution according to the sequence length. The text hidden representation sequence and the noise training data are input into a preset generator, which outputs a predicted duration sequence training data. The text hidden representation sequence is then input into a convolutional discriminator along with the real and predicted duration sequences, respectively, to obtain a first discrimination result (probability 0.92) and a second discrimination result (probability 0.35). The discriminator loss (0.08) and generator loss (0.72) are calculated based on the two types of results. It is then determined whether the discriminator loss (0.08) and generator loss (0.72) meet the preset convergence condition. If not, the model parameters are updated via backpropagation based on the discriminator loss (0.08) and generator loss (0.72). This process is repeated, continuously inputting new financial and medical scenario training data, until the discriminator loss is less than 0.05 and the generator loss is less than 0.1 for five consecutive rounds, satisfying the convergence condition. The generator at this point is the successfully trained adversarial learning-based generator.

[0051] This application's embodiments acquire paired training text and speech data, deriving three types of training data: text hidden representation sequences, real duration sequences, and duration randomized noise. This provides complete input and label samples for adversarial learning training. The generator receives the text hidden representation sequences and noise training data and outputs predicted duration sequences. The discriminator distinguishes between the real and predicted duration sequences, outputting two types of discrimination results. Based on these results, the discriminator loss and generator loss are calculated and backpropagated to update the model parameters. Through iterative training until the convergence condition is met, the generator accurately learns the mapping relationship between text and speech duration. This process replaces the traditional high-complexity streaming model, simplifying the training process while improving duration prediction efficiency.

[0052] In some optional implementations, step S204, based on the text hidden representation sequence and duration randomization noise, uses an adversarial learning-based generator to predict duration, obtaining a predicted duration sequence corresponding to the order of each text unit in the normalized text, specifically including the following steps: By using an adversarial learning-based generator, the text hidden representation sequence and duration randomization noise are fused to obtain preliminary fused features. Then, the adversarial learning-based generator integrates the contextual information and random noise into the preliminary fused features, and outputs the predicted duration sequence corresponding to the order of each text unit in the normalized text.

[0053] Among them, the preliminary fusion feature is the intermediate feature obtained by the generator based on adversarial learning after fusing the input text hidden representation sequence and the duration randomized noise. The fusion method usually includes vector concatenation, element-wise addition or attention weighting, etc. The core function is to initially combine the high-dimensional representation of text semantics with the diverse features of random noise, laying the foundation for further integration of contextual information.

[0054] The integration of contextual information and random noise is a process in which the generator based on adversarial learning performs deep processing on the initial fusion features. This can be achieved through the self-attention mechanism of recurrent neural networks or Transformers. The core is to capture the semantic association (i.e., contextual information) between adjacent text units in the text hidden representation sequence while retaining the random noise in the initial fusion features, so that the final output prediction duration sequence is both diverse and conforms to the temporal logic of text semantics.

[0055] In one example, taking the scenario of a car insurance claim progress notification as an example, the specific implementation process of duration prediction is illustrated. The normalized text to be synthesized is "Your car insurance claim has been accepted, and the claims adjuster will contact you within 2 hours." Its text hidden representation sequence is 10 512-dimensional vectors (corresponding to 10 text units) obtained after processing by a speaker conditional modulation encoder. The duration randomization noise is a 10-dimensional vector sampled from a standard normal distribution, such as [0.12,-0.08,0.15,-0.1,0.09,0.11,-0.07,0.13,-0.09,0.1]. First, the above sequence and noise are fused using a generator based on adversarial learning: each text hidden representation vector is fused with the corresponding noise vector by adding them element-wise, resulting in 10 preliminary 512-dimensional fusion features. These features retain the semantic information of text units such as "car insurance claim" and "claim adjuster" while incorporating the randomness of the noise. Next, the generator integrates contextual information and random noise into the initial fusion features: it captures the semantic association between adjacent text units through the self-attention layer of the Transformer, such as the logical connection between "accepted" and "damage assessor", while retaining noise features through residual connections. Then, it outputs the predicted duration sequence through a fully connected layer: [0.21s, 0.1s, 0.16s, 0.15s, 0.2s, 0.18s, 0.2s, 0.12s, 0.1s, 0.22s]. This sequence corresponds one-to-one with the order of the text units and conforms to the prosodic logic of financial insurance notification voice.

[0056] This application embodiment first fuses the text hidden representation sequence and duration randomization noise using a generator based on adversarial learning. This initially combines the semantic features of text units with random noise, forming a preliminary fused feature that retains both the semantic relevance of the text and random diversity, avoiding the rigidity of duration prediction caused by a single semantic input. Next, the generator integrates contextual information and random noise into the preliminary fused feature, using a network structure to capture the temporal correlation between text units while preserving the diversity brought by the noise. This ensures that the output predicted duration sequence conforms to the contextual logic of the text semantics and avoids overly simplistic duration prediction results. It eliminates the need for a complex stream-based predictor, reducing computational complexity.

[0057] In some optional implementations, step S206, based on Gaussian distribution parameters, target speaker vector, predicted duration sequence, and alignment randomization noise, obtains the alignment relationship between text units and potential speech segments through noisy monotonic alignment search, specifically including the following steps: Speaker feature injection modeling is performed on the normalized text, predicted duration sequence, and target speaker vector to obtain a target speech latent variable sequence. This sequence includes target speech latent variable segments with the same number of text units as the normalized text. Based on the order constraint between the text units of the normalized text and the target speech latent variable segments in the sequence, the range of target speech latent variable segments corresponding to each text unit is determined from the sequence. The index of the target speech segments within this range is not less than the index of the corresponding text unit. Each text unit is then combined with all target speech latent variable segments within its corresponding range. For each combination of text unit and target speech latent variable segments, the mean and variance of the Gaussian distribution parameters are used to determine the optimal combination. The probability logarithm is calculated using a Gaussian distribution function to obtain the matching degree value between the text unit and the target speech latent variable segment corresponding to each combination. Based on a preset monotonically increasing text-speech rule, all local correspondences within the range of the target speech latent variable segment for each text unit are determined, and all local correspondences are connected according to the text unit order of the normalized text to obtain a complete path set containing all text units. Based on a preset alignment initial state score, alignment randomization noise, and matching degree value, the cumulative matching degree of each complete path in the complete path set is calculated using a preset recursive formula for cumulative matching degree with noise constraints. The cumulative matching degree of all complete paths is traversed, and the mapping relationship between the text unit and the target speech latent variable segment corresponding to the complete path with the highest cumulative matching degree is determined as the alignment relationship.

[0058] Among them, the speaker feature injection modeling is a modeling process that fuses the target speaker vector with features such as text and duration to generate a speech latent representation with speaker attributes. It can be achieved through methods such as feature concatenation and conditional layer injection. The core is to make the speech latent variable carry the personalized features of the target speaker.

[0059] Among them, the target speech latent variable sequence is the output result of the speaker feature injection modeling. It is composed of speech latent segments with the same number as the text units. It contains both the speech feature information corresponding to the text semantics and the personalized attributes of the target speaker. It is an intermediate feature carrier between text and speech. The target speech latent variable segment is the basic component unit of the target speech latent variable sequence. Each segment corresponds to the latent representation of a local speech feature, has a semantic association with the text unit, and at the same time carries the features of the target speaker. Its purpose is to serve as an alignment and matching object for text units to achieve a fine-grained association between text and speech. For example, the second segment in the target speech latent variable sequence corresponds to the partial speech features of "day" in "The weather is good today", which is a target speech latent variable segment.

[0060] Among them, the sequential constraint is the rule to be followed when aligning text units with target speech latent variable segments. The core requirement is that the serial number of the speech latent segment is not less than the serial number of the corresponding text unit, so as to ensure the sequential consistency of text semantics and speech flow and avoid the reverse correspondence between speech segments and text units. Its purpose is to constrain the scope of alignment search and ensure that the alignment result conforms to the temporal logic of language expression. For example, the speech latent segment corresponding to the text unit "今" (serial number 1) can only be selected from segments with a serial number greater than or equal to 1, which is a specific manifestation of the sequential constraint.

[0061] Among them, the range of target speech latent variable segments is a set of speech latent segments that each text unit can match, determined based on the sequential constraint. The range boundaries are jointly determined by the text unit serial number and the sequence length. The one-to-one combination is an operation that pairs each text unit with all speech latent segments within the corresponding segment range. The core is to enumerate all possible matching objects of the text unit within the constraint range to provide a complete candidate pairing set for subsequent calculation of the matching degree.

[0062] The Gaussian distribution function is a mathematical function describing the probability distribution of a random variable. Specifically, it refers to the normal distribution function with the mean and variance of the text unit as parameters. Its purpose is to calculate the probabilistic similarity between a text unit and a potential speech segment. The probability logarithm is the result of taking the logarithm of the probability value calculated by the Gaussian distribution function. Its core function is to convert the probability value into a numerical form more suitable for model calculation, while avoiding the loss of computational accuracy caused by low probability values. The matching degree value is a quantitative indicator based on the probability logarithm, representing the degree of matching between the text unit and the potential speech segment. The higher the value, the stronger the semantic and feature association between the two.

[0063] The text-to-speech monotonically increasing rule is a temporal rule that must be followed in the alignment process for local correspondences. It requires that the text unit number and the corresponding speech segment number increase synchronously to avoid reverse correspondences where "the text unit number increases but the speech segment number decreases." Its purpose is to ensure the temporal rationality of local correspondences, providing a foundation for subsequent connection of complete paths. A local correspondence is the pairing relationship between a single text unit and a potential speech segment, generated by the text-to-speech monotonically increasing rule. Its core is the basic unit that constitutes a complete alignment path. The complete path set is the set of all possible alignment paths obtained by connecting the local correspondences of all text units in text order. Its core is that it includes all legal alignment methods between text and speech, providing complete candidates for subsequent selection of the optimal path.

[0064] The initial alignment score is the initial quantized score at the starting position of the alignment path, usually set to a fixed value (such as 0). Its purpose is to provide an initial benchmark for the recursive calculation of the cumulative matching degree. For example, the initial score corresponding to the first text unit and the first segment is set to 0, and the cumulative scores of subsequent paths are recursively calculated from this value. The recursive formula for the cumulative matching degree of a sequence with noise constraints is a mathematical formula for calculating the cumulative matching degree of the complete path. Its core is to introduce alignment randomization noise in the recursive process to increase the flexibility of path search. The formula can be in the form of "current cumulative score = previous score + current matching degree + alignment randomization noise".

[0065] The cumulative matching degree is the sum of the matching degree values ​​of all local correspondences in the complete path calculated by a recursive formula. The higher the value, the better the overall matching degree of the path. It is used as a criterion for selecting the optimal alignment path. The mapping relationship is the set of pairing relationships between all text units and potential speech segments in the complete path. Its core is the final alignment result between text and speech. Its purpose is to establish a precise association between text semantics and speech features.

[0066] In one example, taking the synthesis of the speech "The weather is nice today" of the target speaker "User B" as an example, the process of determining the alignment relationship of the present application will be described in detail. First, preprocess the text to be synthesized "The weather is nice today" to obtain a normalized text, and split it into a sequence of text units: ["今" (sequence number 1), "天" (sequence number 2), "天" (sequence number 3), "气" (sequence number 4), "好" (sequence number 5)]. At the same time, obtain a 10-second reading audio of "User B" as the reference speech, and obtain the target speaker vector (dimension 256) through speech feature encoding processing. Based on the normalized text and this vector, through the speaker-conditioned text encoder, obtain the text hidden representation (dimension 512) and Gaussian distribution parameters of each text unit. For example, for "今", the mean μ1 = 0.5 and the variance σ1 = 0.12; for "天", the mean μ2 = 0.6 and the variance σ2 = 0.15.

[0067] Subsequently, based on the sequence length (5) of the text hidden representation sequence, sample the duration randomization noise from the standard normal distribution (vector [0.2, -0.1, 0.3, -0.2, 0.1]), and input it into the adversarial learning generator to obtain the predicted duration sequence: [0.2s, 0.15s, 0.15s, 0.1s, 0.2s]. Then sample the alignment randomization noise (vector [-0.1, 0.2, -0.3, 0.1, -0.2]) based on this sequence length. Next, perform speaker feature injection modeling, fuse the normalized text, the predicted duration sequence and the target speaker vector, and generate a target speech latent variable sequence (including 5 target speech latent variable segments z1 - z5). Based on the sequential constraint (the sequence number of the speech segment is greater than or equal to the sequence number of the text unit), determine the segment range of each text unit: "今" (sequence number 1) corresponds to [z1 - z5], "天" (sequence number 2) corresponds to [z2 - z5], and so on. Combine each text unit with the segments in the corresponding range one by one (for example, "今" combined with z1 - z5 gets 5 groups of pairings), and calculate the probability logarithm value based on the Gaussian distribution parameters. For example, for the combination of "今" and z1, substitute it into to get -1.8, and this value is the matching degree value. Based on the rule of monotonic increase of text and speech, determine the local correspondence relationship, such as "今" - z1, "天" - z2, etc., and connect them to obtain a complete set of paths. For example, path 1: "今" - z1, "天" - z2, "天" - z3, "气" - z4, "好" - z5; path 2: "今" - z1, "天" - z2, "天" - z4, "气" - z5, "好" - z5. Taking the alignment initial state score (0) as the benchmark, substitute it into the recurrence formula with noise constraints (ε is the alignment randomization noise), calculate the cumulative matching degree of each path: the score of path 1 is -5.2, and the score of path 2 is -6.8. After traversing, select path 1 with the largest cumulative matching degree, and its corresponding mapping relationship is the alignment relationship.

[0068] This application's embodiments utilize speaker feature injection modeling to fuse normalized text, predicted duration sequences, and target speaker vectors to generate a target speech latent variable sequence. This preserves text semantics while incorporating speaker features, avoiding the limitations of manually added intermediate features. Sequence constraints determine the segment range corresponding to each text unit, narrowing the alignment search space while ensuring temporal rationality. After one-to-one combination, matching scores are calculated based on Gaussian distribution parameters, accurately quantifying the correlation between text and speech segments. A complete path set is generated by combining text and speech with monotonically increasing rules, and then the cumulative matching score is calculated using a recursive formula with noise constraints. The introduction of noise enhances the flexibility of path search. Finally, the path with the highest cumulative matching score is selected as the alignment relationship. This avoids the cumbersome process of expanding the input with "whitespace" and achieves precise temporal alignment between text and speech, providing a reliable mapping foundation for subsequent fully end-to-end speech synthesis.

[0069] In some alternative implementations, the step "modeling speaker feature injection from normalized text, predicted duration sequence, and target speaker vector to obtain target speech latent variable sequence" specifically includes the following steps: Based on normalized text and predicted duration sequence, a baseline speech latent variable sequence is generated through a text-to-speech generation model. The baseline speech latent variable sequence and the target speaker vector are fused to obtain the target speech latent variable sequence, which includes target speech latent variable segments with the same number of text units as the normalized text.

[0070] Among them, the text-to-speech generation model is a model used to convert text semantics and duration information into speech latent feature representations. It is usually built based on deep neural networks. Its core function is to receive the semantic features corresponding to normalized text, such as text unit sequences and predicted duration sequences, and generate a baseline speech latent representation that does not carry a specific speaker style by modeling the mapping relationship between text semantics and speech features.

[0071] The baseline speech latent variable sequence is the output of the text-to-speech generation model. It consists of latent variable segments with the same number of text units as the normalized text. Its core representation is the general speech structure features corresponding to the text semantics, such as the temporal framework and basic phoneme features, but it does not include the personalized attributes of a specific target speaker, such as timbre and unique prosody. Its purpose is to serve as a "benchmark carrier" for speaker feature fusion. Through subsequent fusion with the target speaker vector, the final speech latent variable sequence simultaneously possesses the speech structure corresponding to the text semantics and the personalized style of the target speaker.

[0072] In one example, taking a critical illness insurance claim progress notification as an example, the text to be synthesized and normalized is "Your critical illness insurance claim application has been approved," with a corresponding predicted duration sequence of [0.2s, 0.1s, 0.15s, 0.15s, 0.2s, 0.18s, 0.2s]. Both are input into the speech generation model. The model models semantics and speech through mapping, outputting a baseline speech latent variable sequence containing 7 segments, which only contains general speech structure and no specific timbre. The 256-dimensional vector of the target speaker "Insurance Customer Service D" is obtained and fused with each segment of the baseline sequence through attention weighting to obtain the target speech latent variable sequence. This target speech latent variable sequence consists of 7 segments, containing the timbre features of "Customer Service D." Each segment corresponds one-to-one with a text unit, providing a personalized speech carrier for subsequent alignment.

[0073] This application's embodiments, based on normalized text and predicted duration sequences, generate a baseline speech latent variable sequence through a text-to-speech generation model. This model transforms text semantics and duration information into latent representations with general speech structures, such as temporal frames and basic prosody, without relying on manually defined intermediate features, thus avoiding the error accumulation problem in traditional two-stage architectures. The baseline speech latent variable sequence is then fused with the target speaker vector, allowing the generated target speech latent variable sequence to retain the speech structure logic corresponding to the text semantics while injecting the personalized features of the target speaker, such as timbre and unique prosody. Furthermore, the number of segments in the target speech latent variable sequence is consistent with the number of text units, laying the foundation for accurate alignment of subsequent text and speech segments. This eliminates the need to introduce "whitespace" to expand the input, simplifying the processing flow and facilitating fully end-to-end synthesis.

[0074] In some optional implementations, step 207, based on the text hiding representation sequence, alignment relation, and target speaker vector, uses a preset normalized flow model to generate synthesized speech that conforms to the characteristics of the target speaker, specifically including the following steps: Based on the fusion features of the text hidden representation sequence, the speech latent variable segments corresponding to the alignment relationship, and the target speaker vector, a convolutional layer of a pre-defined normalized flow model is used to extract local acoustic details, resulting in a local acoustic feature sequence. Based on the local acoustic feature sequence, a global association layer of attention features in the normalized flow model is used to capture long-distance dependency information, resulting in a global feature sequence. Through residual connection operations, the local acoustic feature sequence and the global feature sequence are fused to obtain the target fused feature sequence. Based on the target fused feature sequence, a decoding layer of the normalized flow model is used to generate synthesized speech that conforms to the characteristics of the target speaker.

[0075] The convolutional layer, a core network layer in the predefined normalized flow model, is used to extract local acoustic details. It can consist of multiple convolutional kernels. Its core function is to extract features within a local window from the fused features of the input, such as text hidden representations, speech latent variable segments, and the target speaker vector. This captures short-term acoustic details in the speech signal, such as pitch, formants, and short-term amplitude variations. The local acoustic feature sequence is the output of the convolutional layer, formed by multiple local acoustic feature vectors arranged in chronological order. Each vector corresponds to the local acoustic properties of a short segment of speech, such as frequency components and energy distribution at a specific moment. It essentially represents the fine-grained acoustic details of the speech signal while preserving its association with text units and speaker features.

[0076] The capture of long-range dependency information is the core function of the global association layer of attention features in the normalized flow model. Through self-attention mechanisms, such as multi-head self-attention, it calculates the association weights between features at any two time steps in the input sequence, capturing long-range acoustic dependencies across short time steps in the speech signal, such as prosodic cohesion and intonation continuity between syllables in a sentence. The global feature sequence is the output of the global association layer of attention features, formed by multiple global feature vectors arranged in chronological order. Each vector not only contains local acoustic information of the corresponding short time step but also incorporates long-range dependency information from other time steps in the sequence, fundamentally representing the global acoustic structure of the speech signal, such as overall intonation trends and prosodic continuity.

[0077] The residual connection operation is a specific operation that fuses local acoustic feature sequences with global feature sequences. It involves directly adding the feature vectors at corresponding time steps of the two sequences, or adding them after weight adjustment, while preserving the original feature information. Its core function is to avoid the vanishing gradient problem in deep network training, while ensuring that the fused features retain both the fine-grained local acoustic details extracted by the convolutional layers and the global long-range dependency information captured by the attention layers. The target fused feature sequence is the output of the residual connection operation. It is composed of feature vectors formed by the residual fusion of the local and global feature sequences, arranged in chronological order. It combines the fine-grained local acoustic details with the coherence of the global acoustic structure, and carries textual semantic information and the target speaker's personalized features, such as timbre and unique intonation. The decoding layer is the core network layer in the normalized flow model used to convert the target fused feature sequence into the final synthesized speech. It is usually built based on reversible transformations. Its core function is to map the high-dimensional target fused feature vector sequence into a continuous speech waveform signal, while preserving the target speaker's personalized features (timbre, intonation) and the speech expression logic corresponding to the textual semantics.

[0078] In one example, taking the medical insurance claim arrival notice as an example, the text corresponding to the content to be synthesized is a hidden representation sequence of text, 5 aligned speech latent variable segments, including semantic association features of "claim arrival", and a 256-dimensional vector of the target speaker "Customer Service E". After the three are fused, they are input into the normalizing flow model. First, the convolutional layer of the normalizing flow model extracts local acoustic details, obtaining a local acoustic feature sequence containing 20 short-time step features, capturing details such as the fundamental frequency (250Hz) of the character "claim" and the short-time energy of the character "payment". Next, the attention feature global association layer of the normalizing flow model calculates the association weights of each short-time step feature through multi-head self-attention, captures the intonation connection between "claim payment" and "arrival", and generates a global feature sequence. Then, the two types of sequences are fused through residual connection to obtain the target fusion feature sequence. Finally, the decoding layer of the normalizing flow model maps this sequence to a 16kHz PCM format audio through a reversible transformation, generating a synthesized speech with the gentle tone of "Customer Service E" and clear semantics of "medical insurance claim arrival", adapting to the speech requirements of the insurance notice scenario.

[0079] In the embodiment of this application, by fusing the text hidden representation sequence, the aligned speech latent variable segments and the target speaker vector, and inputting them into the convolutional layer of the normalizing flow model, the local acoustic details of the speech can be accurately extracted, forming a local acoustic feature sequence that retains fine-grained features, avoiding the problem that traditional models ignore local acoustic information. By capturing long-distance dependence information through the attention feature global association layer, the acoustic features across time steps can be associated, obtaining a global feature sequence with global prosodic coherence, and solving the defect that local features lack overall logic. Through residual connection to fuse the two types of sequences, both local details and global coherence are retained. Finally, the decoding layer maps the target fusion feature sequence to a speech waveform, without artificial intermediate feature and phoneme conversion, achieving complete end-to-end synthesis, while ensuring that the synthesized speech fits the characteristics of the target speaker, improving the naturalness and style matching degree of the speech.

[0080] It should be emphasized that to further ensure the privacy and security of the above continuous normalized text, reference speech, text hidden representation, Gaussian distribution parameters, duration randomization noise, predicted duration sequence, alignment randomization noise, alignment relationship, and synthesized speech, the above normalized text, reference speech, text hidden representation, Gaussian distribution parameters, duration randomization noise, predicted duration sequence, alignment randomization noise, alignment relationship, and synthesized speech can also be stored in a node of a blockchain.

[0081] The blockchain referred to in this application is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0082] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0083] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0084] Further reference Figure 3 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a text-to-speech synthesis apparatus, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0085] like Figure 3 As shown, the text-to-speech synthesis device 400 of this embodiment includes: a receiving module 401, a first generation module 402, a first sampling module 403, a prediction module 404, a second sampling module 405, an alignment module 406, and a second generation module 407. Wherein: The first acquisition module 401 is used to acquire the normalized text of the speech to be synthesized and the reference speech of the target speaker. The first generation module 402 is used to perform speech feature encoding processing on the reference speech, generate the target speaker vector, and obtain the text hidden representation and Gaussian distribution parameters of each text unit of the normalized text based on the normalized text and the target speaker vector. The first sampling module 403 is used to sample from a standard normal distribution based on the sequence length of the text hidden representation sequence to obtain duration randomized noise. The text hidden representation sequence is composed of all text hidden representations. The prediction module 404 is used to predict the duration based on the text hidden representation sequence and duration randomization noise, using an adversarial learning-based generator to obtain the predicted duration sequence corresponding to the order of each text unit of the normalized text. The second sampling module 405 is used to sample from the standard normal distribution based on the sequence length of the predicted duration sequence to obtain aligned randomized noise; Alignment module 406 is used to obtain the alignment relationship between text units and potential speech segments through noisy monotonic alignment search based on Gaussian distribution parameters, target speaker vector, prediction duration sequence and alignment randomization noise. The second generation module 407 is used to generate synthesized speech that conforms to the characteristics of the target speaker based on the text hiding representation sequence, alignment relationship, and target speaker vector, using a preset normalized flow model.

[0086] This application's embodiments utilize an adversarial learning generator combined with duration-randomized noise for duration prediction, replacing traditional stream-based random duration predictors. This reduces the computational complexity and training difficulty of duration prediction, while improving its efficiency. Furthermore, by using normalized text as input, it eliminates the need to convert the text into a phoneme sequence or introduce whitespace to expand the input. Precise alignment between text units and potential speech segments is achieved through noisy monotonic alignment search. Combined with a normalized stream model, the target speech is directly generated, realizing complete end-to-end synthesis from normalized text to speech waveforms. This effectively avoids error accumulation and additional computational overhead caused by intermediate features, ensuring the quality of the synthesized speech.

[0087] In one embodiment, the first generation module 402 includes: The splitting submodule is used to split normalized text into multiple text units according to preset splitting rules. The split text units are then arranged into a text unit sequence according to the text order of the normalized text. The first fusion submodule is used to fuse the text unit sequence with the target speaker vector to obtain the text input with fused speaker features; The encoding mapping submodule is used to encode and map the text input using a speaker-conditional text encoder, generating a text hidden representation for each text unit of the normalized text, as well as the Gaussian distribution parameters of the speech features for each text unit.

[0088] In one embodiment, the prediction module 404 includes: The processing submodule is used to fuse the text hidden representation sequence and duration randomization noise through an adversarial learning-based generator to obtain preliminary fused features; The integration submodule is used to integrate the contextual information and random noise of the preliminary fusion features using an adversarial learning-based generator, and output the predicted duration sequence corresponding to the order of each text unit in the normalized text.

[0089] In one embodiment, the alignment module 406 includes: The modeling submodule is used to perform speaker feature injection modeling on the normalized text, the predicted duration sequence, and the target speaker vector to obtain the target speech latent variable sequence, which includes target speech latent variable segments with the same number of text units as the normalized text. The determination submodule is used to determine the order constraints between text units of normalized text and target speech latent variable segments of the target speech latent variable sequence. It determines the range of target speech latent variable segments corresponding to each text unit from the target speech latent variable sequence, and the index of the target speech segment in the range is not less than the index of the corresponding text unit. The combination submodule is used to combine each text unit with all target speech latent variable segments within the corresponding target speech latent variable segment range one by one; The first calculation submodule is used to calculate the probability logarithm based on the mean and variance in the Gaussian distribution parameters for each combination of text unit and target speech latent variable segment, and obtain the matching degree value between the text unit and the target speech latent variable segment corresponding to each combination. The connecting submodule is used to determine all local correspondences of each text unit corresponding to the target speech latent variable segment range based on the preset text-speech monotonically increasing rule, and to connect all local correspondences according to the text unit order of the normalized text to obtain a complete path set containing all text units; The calculation submodule is used to calculate the cumulative matching degree of each complete path in the complete path set based on the preset alignment initial state score, alignment randomization noise and matching degree value, and through the preset sequence cumulative matching degree recursive formula with noise constraints. The second submodule is used to traverse the cumulative matching degree of all complete paths and determine the mapping relationship between the text unit corresponding to the complete path with the highest cumulative matching degree and the target speech latent variable segment as the alignment relationship.

[0090] In one embodiment, the modeling submodule is further configured to generate a baseline speech latent variable sequence based on normalized text and a predicted duration sequence, using a text-to-speech generation model. The target speech latent variable sequence is obtained by fusing the baseline speech latent variable sequence and the target speaker vector. The target speech latent variable sequence includes target speech latent variable segments with the same number of text units as the normalized text.

[0091] In one embodiment, the second generation module 407 includes: The extraction submodule is used to extract local acoustic details based on the fusion features of the text hidden representation sequence, the speech latent variable segments corresponding to the alignment relationship, and the target speaker vector, and to obtain the local acoustic feature sequence by using the convolutional layer of the pre-defined normalized flow model. The capture submodule is used to capture long-distance dependency information based on the local acoustic feature sequence and the attention feature global association layer of the normalized flow model to obtain the global feature sequence. The second fusion submodule is used to fuse the local acoustic feature sequence with the global feature sequence through residual connection operation to obtain the target fused feature sequence; The generation submodule is used to generate synthesized speech that conforms to the characteristics of the target speaker by using a normalized flow model decoding layer based on the target fusion feature sequence.

[0092] In one embodiment, the text-to-speech synthesis apparatus 400 further includes: The second acquisition module is used to acquire the text and speech data to be trained, and based on the text and speech data to be trained, obtain text hidden representation sequence training data, real duration sequence training data and duration randomized noise training data. The first input module is used to input the text hiding representation sequence training data and the duration randomized noise training data into a preset generator to obtain the predicted duration sequence training data. The second input module is used to input the text-hidden representation sequence training data and the real duration sequence training data into a preset discriminator to obtain the first discrimination result; The third input module is used to input the training data of the text hidden representation sequence and the training data of the prediction duration sequence into the discriminator to obtain the second discrimination result; The calculation module is used to calculate the discriminator loss and the generator loss based on the first and second discrimination results; The update module is used to determine whether the discriminator loss and generator loss meet the preset convergence conditions. If not, the parameters of the discriminator and generator are updated by backpropagation based on the discriminator loss and generator loss, respectively. The execution module is used to acquire new training text and speech data, repeatedly execute the steps of generating new predicted duration sequence training data through the generator for discrimination, and discriminating new real duration sequence training data through the discriminator, until the discriminator loss and generator loss of the new round meet the convergence condition, and the generator corresponding to the convergence condition is determined as the generator based on adversarial learning.

[0093] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0094] Computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected via a system bus. It should be noted that only computer device 6 with memory 61, processor 62, and network interface 63 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0095] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0096] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 6. Of course, the memory 61 may include both the internal storage unit and the external storage device of the computer device 6. In this embodiment, the memory 61 is typically used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions for text-to-speech synthesis methods. In addition, the memory 61 may also be used to temporarily store various types of data that have been output or will be output.

[0097] In some embodiments, processor 62 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 62 is typically used to control the overall operation of the computer device 6. In this embodiment, processor 62 is used to execute computer-readable instructions stored in memory 61 or to process data, such as computer-readable instructions for executing a text-to-speech synthesis method.

[0098] The network interface 63 may include a wireless network interface or a wired network interface, which is typically used to establish a communication connection between the computer device 6 and other electronic devices.

[0099] This application's embodiments utilize an adversarial learning generator combined with duration-randomized noise for duration prediction, replacing traditional stream-based random duration predictors. This reduces the computational complexity and training difficulty of duration prediction, while improving its efficiency. Furthermore, by using normalized text as input, it eliminates the need to convert the text into a phoneme sequence or introduce whitespace to expand the input. Precise alignment between text units and potential speech segments is achieved through noisy monotonic alignment search. Combined with a normalized stream model, the target speech is directly generated, realizing complete end-to-end synthesis from normalized text to speech waveforms. This effectively avoids error accumulation and additional computational overhead caused by intermediate features, ensuring the quality of the synthesized speech.

[0100] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the text-to-speech synthesis method described above.

[0101] This application's embodiments utilize an adversarial learning generator combined with duration-randomized noise for duration prediction, replacing traditional stream-based random duration predictors. This reduces the computational complexity and training difficulty of duration prediction, while improving its efficiency. Furthermore, by using normalized text as input, it eliminates the need to convert the text into a phoneme sequence or introduce whitespace to expand the input. Precise alignment between text units and potential speech segments is achieved through noisy monotonic alignment search. Combined with a normalized stream model, the target speech is directly generated, realizing complete end-to-end synthesis from normalized text to speech waveforms. This effectively avoids error accumulation and additional computational overhead caused by intermediate features, ensuring the quality of the synthesized speech.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of the various embodiments of this application.

[0103] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

[0104] The software tools or components not belonging to our company that appear in the embodiments of this application are merely examples and do not represent actual use.

Claims

1. A text-to-speech synthesis method, characterized in that, Includes the following steps: Obtain the normalized text of the speech to be synthesized and the reference speech of the target speaker; The reference speech is processed by speech feature encoding to generate a target speaker vector. Based on the normalized text and the target speaker vector, the text hidden representation and Gaussian distribution parameters of each text unit of the normalized text are obtained. Based on the sequence length of the text hiding representation sequence, sampling is performed from a standard normal distribution to obtain duration-randomized noise. The text hiding representation sequence is composed of all the text hiding representations. Based on the text hidden representation sequence and the duration randomization noise, a generator based on adversarial learning is used to predict the duration, thereby obtaining a predicted duration sequence corresponding to the order of each text unit of the normalized text. Based on the sequence length of the predicted duration sequence, aligned randomized noise is obtained by sampling from the standard normal distribution; Based on the Gaussian distribution parameters, the target speaker vector, the predicted duration sequence, and the alignment randomization noise, the alignment relationship between text units and potential speech segments is obtained through noisy monotonic alignment search. Based on the text hidden representation sequence, the alignment relationship, and the target speaker vector, a pre-defined normalized flow model is used to generate synthesized speech that conforms to the characteristics of the target speaker.

2. The method according to claim 1, characterized in that, The step of obtaining the text hidden representation and Gaussian distribution parameters of each text unit of the normalized text based on the normalized text and the target speaker vector specifically includes: The normalized text is divided into multiple text units according to a preset splitting rule, and the multiple text units after splitting are arranged into a text unit sequence according to the text order of the normalized text. The text unit sequence is fused with the target speaker vector to obtain a text input with fused speaker features; A speaker-conditional text encoder is used to encode and map the text input, generating a text hidden representation corresponding to each text unit of the normalized text, as well as Gaussian distribution parameters of the speech features corresponding to each text unit.

3. The method according to claim 1, characterized in that, Before the step of using an adversarial learning-based generator to predict the predicted duration sequence corresponding to the order of each text unit in the normalized text based on the text hidden representation sequence and the duration randomization noise, the method further includes: Obtain the text and speech data to be trained, and based on the text and speech data to be trained, obtain text hidden representation sequence training data, real duration sequence training data, and duration randomized noise training data; The text hiding representation sequence training data and the duration randomized noise training data are input into a preset generator to obtain the predicted duration sequence training data; The text-hidden representation sequence training data and the real duration sequence training data are input into a preset discriminator to obtain a first discrimination result; The text hiding representation sequence training data and the predicted duration sequence training data are input into the discriminator to obtain a second discrimination result; Based on the first and second discrimination results, calculate the discriminator loss and the generator loss; Determine whether the discriminator loss and the generator loss satisfy a preset convergence condition. If not, backpropagate and update the parameters of the discriminator and the generator based on the discriminator loss and the generator loss, respectively. New training text and speech data are acquired, and the steps of generating new predicted duration sequence training data through the generator and discriminating it, and discriminating the new real duration sequence training data through the discriminator are repeated until the discriminator loss and generator loss of the new round meet the convergence condition. The generator corresponding to the convergence condition is determined as the generator based on adversarial learning.

4. The method according to claim 3, characterized in that, The step of using an adversarial learning-based generator to predict duration based on the text hidden representation sequence and the duration randomization noise, to obtain a predicted duration sequence corresponding to the order of each text unit in the normalized text, specifically includes: The adversarial learning-based generator is used to fuse the text hidden representation sequence and the duration randomized noise to obtain preliminary fused features. The generator based on adversarial learning is used to integrate contextual information and random noise into the preliminary fusion features, and outputs a predicted duration sequence corresponding to the order of each text unit in the normalized text.

5. The method according to claim 1, characterized in that, The step of obtaining the alignment relationship between text units and potential speech segments through noisy monotonic alignment search based on the Gaussian distribution parameters, the target speaker vector, the predicted duration sequence, and the alignment randomization noise specifically includes: Speaker feature injection modeling is performed on the normalized text, the predicted duration sequence, and the target speaker vector to obtain a target speech latent variable sequence, which includes target speech latent variable segments with the same number of text units as the normalized text. Based on the order constraints between the text units of the normalized text and the target speech latent variable segments of the target speech latent variable sequence, the range of target speech latent variable segments corresponding to each text unit is determined from the target speech latent variable sequence, wherein the sequence number of the target speech segment in the range is not less than the sequence number of the corresponding text unit. Each text unit is combined one by one with all target speech latent variable segments within the corresponding range of target speech latent variable segments; For each combination of text unit and target speech latent variable segment, based on the mean and variance in the Gaussian distribution parameters, the probability logarithm is calculated using the Gaussian distribution function to obtain the matching degree value between the text unit and the target speech latent variable segment corresponding to each combination. Based on the preset text-speech monotonically increasing rule, all local correspondences of the target speech latent variable segment range corresponding to each text unit are determined, and all local correspondences are connected according to the text unit order of the normalized text to obtain a complete path set containing all text units. Based on the preset initial alignment state score, the alignment randomization noise, and the matching degree value, the cumulative matching degree of each complete path in the complete path set is calculated using a preset recursive formula for cumulative matching degree of sequence with noise constraints. Iterate through the cumulative matching degree of all complete paths, and determine the mapping relationship between the text unit corresponding to the complete path with the highest cumulative matching degree and the latent variable segment of the target speech as the alignment relationship.

6. The method according to claim 5, characterized in that, The step of performing speaker feature injection modeling on the normalized text, the predicted duration sequence, and the target speaker vector to obtain the target speech latent variable sequence specifically includes: Based on the normalized text and the predicted duration sequence, a baseline speech latent variable sequence is generated using a text-to-speech generation model. The reference speech latent variable sequence and the target speaker vector are fused to obtain the target speech latent variable sequence, which includes target speech latent variable segments with the same number of text units as the normalized text.

7. The method according to claim 1, characterized in that, The step of generating synthesized speech that conforms to the characteristics of the target speaker based on the text hidden representation sequence, the alignment relationship, and the target speaker vector, using a preset normalized flow model, specifically includes: Based on the fusion features of the text hidden representation sequence, the speech latent variable segments corresponding to the alignment relationship, and the target speaker vector, a local acoustic detail is extracted using a pre-defined normalized flow model convolutional layer to obtain a local acoustic feature sequence. Based on the local acoustic feature sequence, the attention feature global association layer of the normalized flow model is used to capture long-distance dependency information to obtain the global feature sequence. The local acoustic feature sequence is fused with the global feature sequence through a residual connection operation to obtain the target fused feature sequence. Based on the target fusion feature sequence, the normalized flow model's decoding layer is used to generate synthesized speech that conforms to the target speaker's features.

8. A text-to-speech synthesis device, characterized in that, include: The first acquisition module is used to acquire the normalized text of the speech to be synthesized and the reference speech of the target speaker; The first generation module is used to perform speech feature encoding processing on the reference speech, generate a target speaker vector, and obtain the text hidden representation and Gaussian distribution parameters of each text unit of the normalized text based on the normalized text and the target speaker vector. The first sampling module is used to sample from a standard normal distribution based on the sequence length of the text hidden representation sequence to obtain duration randomized noise, wherein the text hidden representation sequence is composed of all the text hidden representations; The prediction module is used to predict the duration based on the text hidden representation sequence and the duration randomization noise, using an adversarial learning-based generator to obtain a predicted duration sequence corresponding to the order of each text unit of the normalized text. The second sampling module is used to sample from the standard normal distribution based on the sequence length of the predicted duration sequence to obtain aligned randomized noise; The alignment module is used to obtain the alignment relationship between text units and potential speech segments through noisy monotonic alignment search based on the Gaussian distribution parameters, the target speaker vector, the predicted duration sequence, and the alignment randomization noise. The second generation module is used to generate synthesized speech that conforms to the characteristics of the target speaker based on the text hidden representation sequence, the alignment relationship, and the target speaker vector, using a preset normalized flow model.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the text-to-speech synthesis method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the text-to-speech synthesis method as described in any one of claims 1 to 7.

Images