Speech synthesis method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to the technical field of medical health and finance, and discloses a speech synthesis method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a to-be-processed text, interpolated noise and unmasked voice; inputting the to-be-processed text, the interpolated noise and the unmasked voice into a synthesis model, coding the to-be-processed text by the synthesis model, combining the adjusted noise and the unmasked voice, learning feature change in a time dimension, converting the noise into a target voice feature, and forming a prediction result; and outputting the prediction result. By implementing the method provided by the embodiment of the invention, the operation efficiency of the zero sample TTS system can be remarkably improved, the deployment cost can be reduced, and the alignment stability and the natural fluency of voice are enhanced while the tone consistency and the content fidelity are kept.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical fields of artificial intelligence, medical health and financial technology, and more particularly to a speech synthesis method and device, a computer device and a storage medium. BACKGROUND

[0002] Current zero-shot TTS (Text-to-Speech) systems, especially those based on diffusion-like models such as Grad-TTS, NaturalSpeech2, have achieved high levels of quality and consistency in voice and tone. However, these systems generally face some significant limitations. The main problems include slow inference speed, high deployment cost, and large model parameters, which make them difficult to be applied in practical scenarios on a large scale. In addition, existing non-autoregressive diffusion / flow matching TTS models usually rely on padding tokens or explicit alignment algorithms to achieve accurate correspondence between text and speech. Although this method simplifies the architecture design, it often leads to inaccurate alignment, reduced speech intelligibility, and slow convergence during the training process.

[0003] For example, models represented by E2-TTS attempt to implicitly learn how to align by expanding text tokens to match the length of the speech. However, this approach is prone to alignment errors, which can affect the quality of the final output. On the other hand, F5-TTS, although it introduces ConvNeXt modules to improve the refinement process of text conditions, still fails to find an ideal balance between few-step inference and maintaining high-quality speech.

[0004] In the medical health scenario, fast and accurate information transmission is crucial. The communication efficiency between doctors and patients directly affects the treatment effect. The use of high-performance TTS technology can improve the reading experience of electronic medical records, help visually impaired patients understand complex medical information, or provide auxiliary communication tools for patients with language barriers. However, the problems mentioned above in the TTS system may limit its application in the medical environment, such as real-time feedback delay due to slow inference speed, or difficulty in deploying on mobile devices due to large model size. For the financial services industry, personalized services and customer experience are key competitive advantages. Using advanced TTS technology can create personalized customer service robots and improve user interaction experience. However, financial services require high security and accuracy, and any speech recognition error can have serious consequences. Therefore, when choosing a TTS solution, the stability and accuracy of the model must be considered. The challenges such as inaccurate alignment and low intelligibility faced by current TTS systems may affect user experience and even cause misunderstandings, especially when dealing with complex financial product introductions or customer service conversations.

[0005] Therefore, it is necessary to design a new method to significantly improve the running efficiency of the zero-shot TTS system and reduce the deployment cost, while maintaining the consistency of timbre and content fidelity, and enhancing the stability of alignment and the natural fluency of speech. SUMMARY

[0006] The present application aims to overcome the defects of the prior art and provide a speech synthesis method and device, a computer equipment and a storage medium.

[0007] To achieve the above object, the present application adopts the following technical scheme: a speech synthesis method, comprising:

[0008] Obtaining a to-be-processed text, an interpolated noise, and an unmasked speech;

[0009] Inputting the to-be-processed text, the interpolated noise, and the unmasked speech into a synthesis model, learning the feature changes in the time dimension by encoding the to-be-processed text and combining the adjusted noise and the unmasked speech through the synthesis model, and converting the noise into target speech features to form a prediction result;

[0010] Outputting the prediction result.

[0011] Further technical solutions thereof are that the synthesis model comprises a text encoder and a flow matching decoder, the text encoder is used for encoding the to-be-processed text, and the flow matching decoder is used for predicting a masked speech in combination with the encoded text, the interpolated noise, and the unmasked speech.

[0012] Further technical solutions thereof are that the training process of the synthesis model comprises:

[0013] The teacher model is used to guide the learning process of the student model through double-step reasoning, and the student model parameters are updated by using an exponential moving average method to continue the training process of the synthesis model.

[0014] Further technical solutions thereof are that the training process of the synthesis model comprises:

[0015] The model composed of the text encoder and the flow matching decoder is trained by using a dynamic CFG.

[0016] Further technical solutions thereof are that the inputting the to-be-processed text, the interpolated noise, and the unmasked speech into the synthesis model, learning the feature changes in the time dimension by encoding the to-be-processed text and combining the adjusted noise and the unmasked speech through the synthesis model, and converting the noise into target speech features to form a prediction result comprises:

[0017] The text encoder receives the to-be-processed text and converts the to-be-processed text into an encoding capable of representing the text context and semantics to obtain an encoding result.

[0018] The flow matching decoder combines the encoding result, the interpolated noise and the unmasked speech to predict the masked speech to obtain a prediction result.

[0019] Further technical solutions of the present application are as follows: the text encoder receives the to-be-processed text and converts the to-be-processed text into an encoding capable of representing the text context and semantics to obtain an encoding result, including:

[0020] The text encoder adopting a Zipformer architecture receives the to-be-processed text and converts the to-be-processed text into an encoding capable of representing the text context and semantics to obtain an encoding result; wherein the text encoder comprises a U-Net structure, a convolution module and an attention weight reuse mechanism.

[0021] Further technical solutions of the present application are as follows: the flow matching decoder combines the encoding result, the interpolated noise and the unmasked speech to predict the masked speech to obtain a prediction result, including:

[0022] The flow matching decoder combines the encoding result, the interpolated noise and the unmasked speech to learn a time-dependent vector field, converts the interpolated noise into a target speech feature distribution by solving an ordinary differential equation to obtain a prediction result.

[0023] The present application also provides a speech synthesis device, comprising:

[0024] An acquisition unit is configured to acquire a to-be-processed text, interpolated noise and unmasked speech.

[0025] A prediction unit is configured to input the to-be-processed text, the interpolated noise and the unmasked speech into a synthesis model, and learn the feature changes in the time dimension by encoding the to-be-processed text and combining the adjusted noise and the unmasked speech, and convert the noise into a target speech feature to form a prediction result.

[0026] An output unit is configured to output the prediction result.

[0027] The present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above method.

[0028] The present application also provides a storage medium storing a computer program, wherein the computer program is executed by a processor to implement the above method.

[0029] The beneficial effects of the present application compared with the prior art are: the present application obtains the to-be-processed text, the interpolated noise and the unmasked speech, inputs them into a synthesis model including a text encoder and a flow matching decoder, performs prediction on the masked speech to obtain a prediction result, and finally outputs the result. This process utilizes advanced deep learning algorithms, such as a text encoder adopting a Zipformer architecture and a flow matching decoder combining a time-dependent vector field to solve ordinary differential equations, which not only significantly improves the running efficiency of the zero-shot TTS system and reduces the deployment cost, but also maintains the consistency of timbre and the fidelity of content. In addition, through an effective training mechanism, the stability of alignment and the natural fluency of speech are enhanced, so that high-quality speech output can be generated even without a large number of samples, meeting the needs of practical applications. This method greatly promotes the wide application of TTS technology in multiple scenarios, including but not limited to personalized customer service, intelligent assistants and barrier-free communication fields.

[0030] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0032] Figure 1 Application scenario diagram of the speech synthesis method provided by the embodiment of the present application;

[0033] Figure 2 Flowchart of the speech synthesis method provided by the embodiment of the present application;

[0034] Figure 3 Subflowchart of the speech synthesis method provided by the embodiment of the present application;

[0035] Figure 4 Schematic block diagram of the speech synthesis device provided by the embodiment of the present application;

[0036] Figure 5 Schematic block diagram of the prediction unit of the speech synthesis device provided by the embodiment of the present application;

[0037] Figure 6 Schematic block diagram of the computer device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0038] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of the present application.

[0039] It should be understood that the terms "comprising" and "including" as used in the specification and the appended claims indicate the presence of the described features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0040] It should also be understood that the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0041] It should be further understood that the term "and / or" used in the present application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations thereof, and includes these combinations.

[0042] Please refer to Figure 1 , Figure 1 The present application provides a schematic flowchart of a speech synthesis method. The speech synthesis method is applied to a server. The server interacts with a terminal to perform data interaction, integrates a text encoder based on a Zipformer architecture and a flow matching decoder based on a CFM (Conditional Flow Matching) framework, uses interpolated noise, unmasked speech and context-enhanced semantic encoding as input conditions to perform masked speech prediction, adopts a flow distillation mechanism and a dynamic CFG scheme to optimize the training process, and solves ordinary differential equations to achieve efficient conversion from noise to target speech feature distribution. This series of technologies not only significantly improves the running efficiency of the system and reduces the deployment cost, but also ensures the consistency of the timbre and the fidelity of the content, enhances the stability of the text and speech alignment and the natural fluency of the generated speech, thereby realizing model simplification and performance optimization while maintaining high-quality output.

[0043] Figure 2 is a flowchart of the speech synthesis method provided by the present application. As shown in Figure 2 , the method comprises the following steps S110 to S150.

[0044] S110, obtaining the text to be processed, the interpolated noise, and the unmasked speech.

[0045] In this embodiment, the text to be processed refers to the original text data that has not undergone any processing or conversion, and is usually used in subsequent natural language processing tasks such as text analysis, sentiment calculation, or machine translation.

[0046] Interpolated noise refers to additional noise data points generated between original noise samples through mathematical methods such as linear interpolation or other advanced interpolation techniques, with the purpose of increasing data volume or adjusting noise distribution, commonly used in audio processing and machine learning fields, which helps to improve the robustness of model training.

[0047] Unmasked speech represents the original speech signal that has not been denoised or feature-masked, preserving all its feature information, which can be used for direct analysis of speech quality in real environments or as raw input material for training deep learning models.

[0048] In a text-to-speech (TTS) system, the first step is to obtain the text to be processed. This process usually involves receiving raw text information from user input or other data sources. For a zero-shot TTS system, the key in this stage is how to effectively convert natural language text into a form that the model can understand. This includes but is not limited to word segmentation, lexicalization, and semantic encoding of cultural context.

[0049] By using Zipformer as a lightweight architecture to implement the text encoder, the system not only efficiently captures semantic information in the text, but also significantly reduces the number of model parameters. Specifically, Zipformer introduces a U-Net-like down-sampling and up-sampling structure and convolutional modules that enhance the understanding of local information and reduce the demand for computing resources through attention weight reuse mechanisms. This optimization makes the system more compact while maintaining high performance, suitable for edge device application scenarios.

[0050] During the generation process, the flow matching decoder needs to receive interpolated noise as one of the input conditions. The noise here is obtained by interpolating the initial distribution (such as Gaussian noise) through some algorithm, with the purpose of guiding the model to gradually approach the target speech feature distribution. The interpolation process helps to ensure that the generated speech has sufficient diversity and naturalness.

[0051] Using the CFM framework, the system learns a time-dependent vector field v t (x t; θ), gradually transforming the initial noise into target speech features. Compared to the diffusion model, CFM reduces the number of steps and improves training stability. In addition, combined with the flow distillation mechanism, significant breakthroughs have been made in inference acceleration, requiring only 4 steps of inference to complete high-quality synthesis, while avoiding the double inference cost of CFG.

[0052] Another important input condition is the part of the speech prompt that is not masked. This part of the speech provides key information to help the model better understand and predict the masked part that needs to be reconstructed. It acts as a real speech sample for comparison and adjustment of the generated speech to make it as close to real speech as possible.

[0053] The system uses an average upsampling strategy to simplify the alignment process between text and speech. This method assumes that each text token has equal duration, thereby avoiding complex duration predictors. Despite its simplicity, it performs well in practical applications, significantly improving speech intelligibility and naturalness. In addition, the dynamic CFG scheme further optimizes the inference process of the non-distilled version, ensuring a balance between content and timbre, while the distilled version directly inherits this ability without additional adjustments.

[0054] The above-mentioned improvements in multiple aspects have made significant progress in efficiency, model compactness, robustness, and timbre consistency. In particular, on large-scale datasets, it can still maintain speech quality comparable to the state-of-the-art (SOTA). At the same time, the running speed of the system on GPU and CPU has been improved several times, greatly expanding its application scenario range. In summary, based on the theoretically robust flow matching method, combined with architecture and inference optimization, a high-efficiency, simple, and high-quality zero-shot TTS system is formed, laying a solid foundation for the landing of practical TTS applications.

[0055] S120, input the to-be-processed text, the interpolated noise, and the unmasked speech into the synthesis model, and learn the feature changes in the time dimension by encoding the to-be-processed text and combining the adjusted noise and the unmasked speech, and convert the noise into target speech features to form a prediction result.

[0056] The synthesis model learns the feature changes in the time dimension by encoding the to-be-processed text and combining the adjusted noise and the unmasked speech, and converts the noise into target speech features to form a prediction result.

[0057] In this embodiment, the predicted result refers to the predicted value of the masked speech part generated by the synthesis model, aiming to reconstruct the masked part in the real speech signal. This process utilizes two core components, a text encoder and a flow-matching decoder, as well as an innovative training mechanism to achieve high-quality speech synthesis.

[0058] The synthesis model includes a text encoder for encoding the text to be processed and a flow-matching decoder for predicting the masked speech in combination with the encoded text, interpolated noise, and unmasked speech.

[0059] The synthesis model is trained using a flow distillation mechanism, where a teacher model guides a student model through a two-step inference process, and the student model is trained using an exponential moving average. Specifically, the training process of the synthesis model includes:

[0060] The teacher model guides the learning process of the student model through a two-step inference process, and the student model parameters are updated using an exponential moving average method to continue the training process of the synthesis model

[0061] Specifically, the flow distillation mechanism is a training strategy that transfers the knowledge of a complex teacher model to a relatively simple student model. This method is particularly suitable for scenarios that require efficient inference, such as real-time speech synthesis systems. The following are the specific implementation steps of this mechanism:

[0062] Two-step inference guidance: First, a teacher model with higher performance but possibly more complex or resource-intensive is used for inference. The teacher model provides guidance to the student model through a two-step inference process:

[0063] The first step of inference usually produces preliminary results or intermediate representations.

[0064] The second step further optimizes or refines the output based on the results of the first step.

[0065] This two-step inference approach helps the student model learn more refined and accurate feature representations.

[0066] Exponential moving average student model: During training, instead of directly updating the original student model parameters, an exponential moving average technique is used to smooth the model parameters. Exponential moving average calculates the current best estimate by assigning certain weights to historical parameters, which helps stabilize the training process, avoids overfitting, and enables the student model to better approximate the performance of the teacher model.

[0067] In addition, the synthesis model is trained using dynamic CFG. Specifically, the training process of the synthesis model includes:

[0068] Training with dynamic CFG involves a model composed of a text encoder and a flow matching decoder.

[0069] Specifically, dynamic CFG refers to a flexible adjustment of context-free grammar strategy when training a synthetic model. Traditional CFG is a fixed rule set, while dynamic CFG allows adaptive adjustment of generated rules according to the characteristics of input data, thereby improving the generalization ability and expressiveness of the model. Specifically:

[0070] Dynamic CFG can automatically adjust its internal structure or rule set according to different input samples, so that the model can make the optimal response to specific scenarios or tasks.

[0071] By introducing dynamic CFG, the ability of the speech synthesis model to understand language structure can be enhanced to some extent, especially for languages with complex grammar structures or domain-specific vocabulary.

[0072] This training method can help improve the naturalness and accuracy of speech synthesis, as it allows the model to more carefully consider contextual information when generating speech, resulting in smoother and more coherent voice output.

[0073] In summary, the training method combining flow distillation mechanism and dynamic CFG aims to build an efficient and high-quality speech synthesis system that can guarantee fast and accurate reasoning ability while providing rich and delicate voice performance.

[0074] In an embodiment, referring to Figure 3 The above step S120 can include steps S121-S122.

[0075] S121, using a text encoder to receive the to-be-processed text, and converting the to-be-processed text into an encoding capable of representing the context and semantics of the text to obtain an encoding result.

[0076] Specifically, the text encoder using the Zipformer architecture receives the to-be-processed text and converts the to-be-processed text into an encoding capable of representing the context and semantics of the text to obtain an encoding result; wherein the text encoder includes a U-Net structure, a convolution module and an attention weight reuse mechanism.

[0077] In this embodiment, the encoding result refers to the context-enhanced semantic representation generated by the text encoder after analyzing the input to-be-processed text. Specifically, the text encoder based on the Zipformer architecture receives the to-be-processed text and converts the to-be-processed text into an encoding form capable of effectively representing the context and semantic information of the text. This text encoder has the following characteristics:

[0078] U-Net architecture: This architecture allows the model to effectively capture information at different levels, enhancing the understanding of both local and global features through the process of downsampling and upsampling.

[0079] Convolutional modules: Strengthen the model's ability to capture local information, helping to improve the understanding of detailed information.

[0080] Attention weight reuse mechanism: Reduces the number of parameters and computational complexity while maintaining model performance, improving efficiency.

[0081] Through the above components, the text encoder can extract rich context and semantic information from the input text to be processed, providing a solid foundation for subsequent speech synthesis.

[0082] S122, using a flow matching decoder to combine the encoding results, interpolated noise, and unmasked speech to predict the masked speech, resulting in a prediction result.

[0083] Specifically, using a flow matching decoder to combine the encoding results, interpolated noise, and unmasked speech to learn a time-dependent vector field, and converting the interpolated noise into the target speech feature distribution by solving ordinary differential equations to obtain the prediction result.

[0084] Specifically, in this step, the flow matching decoder receives three main inputs: the encoding results generated by the text encoder (i.e., the context-enhanced semantic representation), the interpolated noise, and the unmasked speech cues. These inputs work together in the decoder, with the goal of predicting the masked part of the speech signal. Here's the specific process:

[0085] Combine multiple inputs: The flow matching decoder first concatenates the three types of input information to form a comprehensive input vector.

[0086] Learn time-dependent vector field: Through the Conditional Flow Matching (CFM) framework, the decoder attempts to learn a time-dependent vector field v t (x t ; θ) that gradually transforms the initial distribution (such as Gaussian noise) into the target speech feature distribution.

[0087] Solve ordinary differential equations (ODEs): To achieve this transformation, the decoder is actually solving one or more ordinary differential equations (ODEs) to gradually approximate the representation of the target speech by iteratively updating the noise samples.

[0088] Ultimately, through such a process, the stream matching decoder is able to generate high-quality masked speech prediction results that closely approximate the corresponding portions of the real speech signal, resulting in accurate and natural speech synthesis.

[0089] In this embodiment, specifically, first, the text to be processed is input into the text encoder for encoding. The encoder, based on a zipformer structure, can effectively extract semantic information in the text and convert it into a context-enhanced semantic representation. In this process, the zipformer adopts a U-Net-like downsampling-upsampling structure, convolutional modules, and attention weight reuse mechanisms, which not only enhance the ability to capture local information but also improve parameter efficiency and reduce computational complexity.

[0090] Next, the encoded text information, interpolated noise, and unmasked speech cues are jointly input into the stream matching decoder. The task of the stream matching decoder is to combine the information of the three to predict the masked portion of the speech signal. To ensure the accuracy of the prediction, the decoder uses the CFM framework to learn a time-dependent vector field, gradually transforming the initial distribution (such as Gaussian noise) into the target speech feature distribution, thereby achieving high-quality speech synthesis.

[0091] In addition, to further improve the inference speed and performance of the model, the synthesis model uses a stream distillation mechanism for training. This mechanism first uses a teacher model (usually a larger and more complex model) to guide the learning of the student model through double-step inference, and then uses the exponential moving average (EMA) student model as the basis for subsequent training, gradually refining the prediction ability of the model. This method not only significantly reduces the number of steps required for inference, but also avoids the additional computational cost of traditional CFG methods, allowing the final model to maintain high-quality speech synthesis while achieving fast inference.

[0092] Finally, the dynamic CFG scheme is also used in the training process of the non-distilled version, by removing the text condition at the beginning and the text and audio conditions at the end, ensuring the consistency of the timbre and the fidelity of the content. The distilled version directly inherits these optimizations, omitting the adjustment process during inference.

[0093] In summary, through the above series of designs and optimizations, the synthesis model not only achieves accurate prediction of the masked speech portion, but also exhibits excellent performance in multiple aspects, including but not limited to significant improvement in inference speed, significant reduction in model size, and good adaptability to different devices, etc., promoting the practical application and development of zero-shot TTS technology.

[0094] S130, output the prediction result.

[0095] The prediction result is output to a terminal for playing, etc.

[0096] In this embodiment, a conventional speech synthesis model usually needs 32 or more inference steps to generate high-quality speech output, while the present scheme reduces the number of inference steps to only 4 by introducing a flow distillation mechanism and a CFM framework. This not only greatly speeds up the speech synthesis, achieving up to 32 times acceleration on CPU, making the system close to real-time operation, but also enables efficient deployment even on resource-constrained edge devices, making it possible for practical applications.

[0097] The zipformer is used as a lightweight architecture, combined with an attention weight reuse mechanism and a U-Net-like down-sampling-up-sampling structure, effectively reducing the number of model parameters, with the entire model size being only 123M, which is about two-thirds smaller than other advanced TTS models such as F5-TTS. This compactness not only reduces the memory overhead, but also reduces the model loading time, improving the system's response speed, which is of great significance for mobile devices and cloud services.

[0098] The application of the average up-sampling strategy simplifies the text-to-speech alignment process, achieving stable and efficient alignment without additional duration predictors. This method, although simple, performs well in actual tests, especially in maintaining speech intelligibility and naturalness. It avoids the complexity and instability problems that may occur in traditional methods, enhancing the overall robustness of the system.

[0099] By combining CFG (Context-Free Grammar) with the flow distillation mechanism, the model can achieve a good balance between voice quality transfer and semantic consistency. This means that whether it is between different speakers or under different recording conditions of the same speaker, the system can maintain high voice consistency and content accuracy, ensuring the quality and realism of the generated speech.

[0100] Overall, based on the above improvements, the present embodiment not only achieves speech quality comparable to SOTA (State-of-the-Art), but also makes important breakthroughs in efficiency, compactness, stability, and practicality, becoming a key step in advancing TTS technology to more widely applicable scenarios. This progress marks a top-level speech synthesis effect without relying on large models, greatly broadening the application range and potential of TTS technology.

[0101] For example, in the financial field, this method can be used to generate personalized financial voice broadcasts in real time. The following information is obtained:

[0102] The text to be processed: daily fund net value fluctuations, macroeconomic news, and user portfolio profit and loss summaries.

[0103] Interpolated noise: Use stock market high-frequency K-line or order book data to do "interpolation noise" to map the statistical characteristics of price fluctuations to the noise vector. This preserves market volatility information and allows the model to learn "denoising" during training.

[0104] Unmasked voice: Keep the complete broadcast segment that the user last listened to as a "known clean segment."

[0105] The model only needs to complete the masked segment (e.g., sudden jump "emergency risk prompt").

[0106] Training mechanism:

[0107] Teacher model: Use the complete broadcast after the past three years' closing to do double-step reasoning to teach the student model how to quickly complete real-time broadcasts.

[0108] Dynamic CFG: Let the model learn the broadcast speed and emotions under different market volatility levels during the training phase (bear market is more stable, bull market is more lively).

[0109] The broker App can generate "read-only new content" voice in milliseconds, saving 70% bandwidth and maintaining the user's familiar anchor tone and emotions.

[0110] Another example, Ms. Zhang was involved in a rear-end collision on the highway, fracturing her left arm. She immediately called the 955** claims reporting hotline of a certain insurance company. Due to unstable tunnel signals, three segments of 0.5-1 seconds of silence appeared in the recording, which swallowed the hospital name and diagnosis code in the key sentence "I am in a certain hospital emergency, diagnosed with left ulna fracture."

[0111] The system immediately starts the patented voice completion process:

[0112] The front-end IVR sends this segment of the recording with silence to the cloud in real time;

[0113] The medical ASR first identifies "I am in a <mask> <mask>;

[0114] With clear speech for 300 ms before and after the blank, tunnel environment noise as the condition, Zipformer flow matching model completes the missing syllable within 150 ms, generating a 48 kHz speech segment of "certain hospital" "left ulna fracture";

[0115] The completed sentence is automatically played back to the customer service headset and labeled with AI, along with a confidence of 0.93, model version v2.1.3, and written into the blockchain evidence chain.

[0116] The customer service does not need to ask again, directly confirms the injury and sends the investigation staff, and the report of Ms. Zhang is completed within 3 minutes; the background statistics show that the telephone call saves 8 minutes of manual re-listening due to AI completion, and the claim time limit is shortened from T+1 to T+0.

[0117] The above-mentioned speech synthesis method, by acquiring the to-be-processed text, the interpolated noise and the unmasked speech, and inputting them into a synthesis model including a text encoder and a flow matching decoder, predicting the masked speech to obtain a prediction result, and finally outputting the result. This process utilizes advanced deep learning algorithms, such as a text encoder using Zipformer architecture and a flow matching decoder combining time-dependent vector field to solve ordinary differential equations, not only significantly improves the running efficiency of zero-shot TTS system and reduces the deployment cost, but also maintains the consistency of tone and the fidelity of content. In addition, through effective training mechanism, the stability of alignment and the natural fluency of speech are enhanced, so that high-quality speech output can be generated even without a large number of samples, meeting the needs of practical applications. This method greatly promotes the wide application of TTS technology in multiple scenarios, including but not limited to personalized customer service, intelligent assistants and barrier-free communication fields.

[0118] Figure 4 is a schematic block diagram of a speech synthesis device 300 provided by an embodiment of the present application. As shown in Figure 4 Corresponding to the above speech synthesis method, the present application also provides a speech synthesis device 300. The speech synthesis device 300 includes units for executing the above speech synthesis method, and the device can be configured in a server. Specifically, please refer to Figure 4 The speech synthesis device 300 includes an acquisition unit 301, a prediction unit 302 and an output unit 303.

[0119] The acquisition unit 301 is configured to acquire a to-be-processed text, interpolated noise, and unmasked speech; the prediction unit 302 is configured to input the to-be-processed text, the interpolated noise, and the unmasked speech into a synthesis model, learn feature changes in a time dimension by encoding the to-be-processed text and combining the adjusted noise and the unmasked speech, and convert the noise into target speech features to form a prediction result; the output unit 303 is configured to output the prediction result.

[0120] In an embodiment, the prediction unit 302 includes an encoding subunit 3021 and a decoding prediction subunit 3022.

[0121] The encoding subunit 3021 is configured to receive the to-be-processed text by using a text encoder, and convert the to-be-processed text into an encoding capable of representing a text context and semantics to obtain an encoding result; the decoding prediction subunit 3022 is configured to perform prediction of masked speech by combining the encoding result, the interpolated noise, and the unmasked speech by using a flow matching decoder to obtain a prediction result.

[0122] In an embodiment, the encoding subunit 3021 is configured to receive the to-be-processed text by using a text encoder of a Zipformer architecture, and convert the to-be-processed text into an encoding capable of representing a text context and semantics to obtain an encoding result; the text encoder includes a U-Net type structure, a convolution module, and an attention weight reuse mechanism.

[0123] In an embodiment, the decoding prediction subunit 3022 is configured to learn a time-dependent vector field by combining the encoding result, the interpolated noise, and the unmasked speech by using a flow matching decoder, convert the interpolated noise into a target speech feature distribution by solving an ordinary differential equation to obtain a prediction result.

[0124] It should be noted that a person skilled in the art can clearly understand the specific implementation process of the above-mentioned speech synthesis device 300 and each unit, which can refer to the corresponding description in the foregoing method embodiments. For the convenience and brevity of description, it will not be repeated here.

[0125] The speech synthesis device 300 described above can be implemented in the form of a computer program, which can run on a computer device as shown in Figure 6 .

[0126] Please refer to Figure 6 , Figure 6 is a schematic block diagram of a computer device provided by an embodiment of the present application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0127] Referring to Figure 6 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected through a system bus 501, wherein the memory can include a non-volatile storage medium 503 and an internal memory 504.

[0128] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a speech synthesis method.

[0129] The processor 502 is configured to provide computing and control capabilities to support the operation of the entire computer device 500.

[0130] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503, which, when executed by the processor 502, causes the processor 502 to perform a speech synthesis method.

[0131] The network interface 505 is configured to perform network communication with other devices. Those skilled in the art can understand that Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device 500 to which the scheme of the present application is applied. The specific computer device 500 can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0132] The processor 502 is configured to run the computer program 5032 stored in the memory to implement the following steps:

[0133] Obtain the to-be-processed text, the interpolated noise, and the unmasked speech; input the to-be-processed text, the interpolated noise, and the unmasked speech into a synthesis model; the synthesis model learns the feature changes in the time dimension by encoding the to-be-processed text and combining the adjusted noise and the unmasked speech, and converts the noise into target speech features to form a prediction result; output the prediction result.

[0134] The synthesis model comprises a text encoder and a stream matching decoder, the text encoder is used for encoding the to-be-processed text, and the stream matching decoder is used for combining the encoded text, the interpolated noise, and the unmasked speech prediction mask speech.

[0135] The training process of the synthesis model comprises:

[0136] The teacher model is used to guide the learning process of the student model through double-step reasoning, and the student model parameters are updated by using an exponential moving average method to continue the training process of the synthesis model.

[0137] The training process of the synthesis model comprises:

[0138] The model composed of the text encoder and the stream matching decoder is trained by using a dynamic CFG. In an embodiment, when the processor 502 implements the step of inputting the to-be-processed text, the interpolated noise, and the unmasked speech into the synthesis model, learning the feature change in the time dimension by encoding the to-be-processed text and combining the adjusted noise and the unmasked speech, and converting the noise into target speech features to form a prediction result, the following steps are specifically implemented:

[0139] The text encoder is used to receive the to-be-processed text and convert the to-be-processed text into an encoding capable of representing the text context and semantics to obtain an encoding result, and the stream matching decoder is used to combine the encoding result, the interpolated noise, and the unmasked speech to predict the masked speech to obtain a prediction result.

[0140] In an embodiment, when the processor 502 implements the step of using the text encoder to receive the to-be-processed text and convert the to-be-processed text into an encoding capable of representing the text context and semantics to obtain an encoding result, the following steps are specifically implemented:

[0141] The text encoder of the Zipformer architecture is used to receive the to-be-processed text and convert the to-be-processed text into an encoding capable of representing the text context and semantics to obtain an encoding result, wherein the text encoder comprises a U-Net type structure, a convolution module, and an attention weight reuse mechanism.

[0142] In an embodiment, when the processor 502 implements the step of using the stream matching decoder to combine the encoding result, the interpolated noise, and the unmasked speech to predict the masked speech to obtain a prediction result, the following steps are specifically implemented:

[0143] The flow matching decoder combines the encoding result, the interpolated noise, and the unmasked speech learning time-dependent vector field to convert the interpolated noise into a target speech feature distribution by solving an ordinary differential equation to obtain a prediction result.

[0144] It should be understood that, in the embodiments of the present application, the processor 502 can be a central processing unit (CPU), and the processor 502 can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0145] It can be understood by those skilled in the art that all or part of the processes in the method of the above-mentioned embodiments can be completed by instructing relevant hardware by a computer program. The computer program includes program instructions, and the computer program can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the above-mentioned embodiments of the method.

[0146] Therefore, the present application also provides a storage medium. The storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein the computer program is executed by a processor to make the processor perform the following steps:

[0147] The text to be processed, the interpolated noise, and the unmasked speech are obtained, and the text to be processed, the interpolated noise, and the unmasked speech are input into a synthesis model. The synthesis model encodes the text to be processed, combines the adjusted noise and the unmasked speech, learns the feature changes in the time dimension, and converts the noise into target speech features to form a prediction result. The synthesis model encodes the text to be processed, combines the adjusted noise and the unmasked speech, learns the feature changes in the time dimension, and converts the noise into target speech features to form a prediction result. The prediction result is output.

[0148] The synthesis model includes a text encoder and a flow matching decoder. The text encoder is used to encode the text to be processed. The flow matching decoder is used to combine the encoded text, the interpolated noise, and the unmasked speech to predict the masked speech.

[0149] The training process of the synthetic model comprises:

[0150] The learning process of the student model is guided by the teacher model through double-step reasoning, and the student model parameters are updated using the exponential moving average method to continue the training process of the synthetic model.

[0151] The training process of the synthetic model comprises:

[0152] The model composed of a text encoder and a flow matching decoder is trained using dynamic CFG.

[0153] In an embodiment, when the processor implements the step of inputting the to-be-processed text, the interpolated noise, and the unmasked speech into the synthetic model, learning the feature changes in the time dimension by encoding the to-be-processed text and combining the adjusted noise and the unmasked speech, and converting the noise into target speech features to form a prediction result, the processor implements the following steps when executing the computer program:

[0154] The text encoder receives the to-be-processed text and converts the to-be-processed text into an encoding that can represent the context and semantics of the text to obtain an encoding result; and the flow matching decoder combines the encoding result, the interpolated noise, and the unmasked speech to predict the masked speech to obtain a prediction result.

[0155] In an embodiment, when the processor implements the step of receiving the to-be-processed text using the text encoder and converting the to-be-processed text into an encoding that can represent the context and semantics of the text to obtain an encoding result, the processor implements the following steps when executing the computer program:

[0156] The text encoder of the Zipformer architecture receives the to-be-processed text and converts the to-be-processed text into an encoding that can represent the context and semantics of the text to obtain an encoding result; wherein the text encoder comprises a U-Net structure, a convolution module, and an attention weight reuse mechanism.

[0157] In an embodiment, when the processor implements the step of combining the encoding result, the interpolated noise, and the unmasked speech using the flow matching decoder to predict the masked speech to obtain a prediction result, the processor implements the following steps when executing the computer program:

[0158] The flow matching decoder combines the encoding result, the interpolated noise, and the unmasked speech to learn a time-dependent vector field, and converts the interpolated noise into a target speech feature distribution by solving an ordinary differential equation to obtain a prediction result.

[0159] The storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, and the like various computer readable storage media that can store program codes.

[0160] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in a general manner. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0161] In several embodiments provided by the present application, it should be understood that the disclosed apparatus and method can be implemented by other ways. For example, the apparatus embodiments described above are merely schematic. For example, the division of each unit is only a logical function division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0162] The steps in the method embodiments of the present application can be adjusted, combined and deleted in sequence according to actual needs. The units in the apparatus embodiments of the present application can be combined, divided and deleted according to actual needs. In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0163] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part of the prior art that makes a contribution, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a terminal or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0164] The above merely illustrates the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any skilled person in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements shall be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.< / mask> < / mask>

Claims

1. A speech synthesis method, characterized in that: include: Get the text to be processed, the interpolated noise, and the unmasked speech; Inputting the text to be processed, the interpolated noise, and the unmasked speech into a synthesis model, the synthesis model encodes the text to be processed, combines the adjusted noise and the unmasked speech, learns feature changes in the time dimension, and converts the noise into target speech features to form a prediction result; The prediction result is output.

2. The speech synthesis method according to claim 1, wherein: The synthesis model includes a text encoder and a stream matching decoder, wherein the text encoder is used to encode the text to be processed; The stream matching decoder is used to combine the encoded text, the interpolated noise, and the unmasked speech to predict the masked speech.

3. The speech synthesis method according to claim 2, wherein: The training process of the synthetic model includes: The teacher model is used to guide the learning process of the student model through two-step reasoning, and the exponential moving average method is used to update the student model parameters to continue the training process of the synthetic model.

4. The speech synthesis method according to claim 2, wherein: The training process of the synthetic model includes: Dynamic CFG is used to train the model consisting of a text encoder and a stream matching decoder.

5. The speech synthesis method according to claim 2, wherein: The text to be processed, the interpolated noise, and the unmasked speech are input into the synthesis model, and the synthesis model encodes the text to be processed and combines the adjusted noise and the unmasked speech to learn the feature changes in the time dimension, and converts the noise into target speech features to form a prediction result, including: Using a text encoder to receive the text to be processed, and converting the text to be processed into a code that can represent the text context and semantics to obtain an encoding result; The masked speech is predicted by using a stream matching decoder in combination with the encoding result, the interpolated noise, and the unmasked speech to obtain a prediction result.

6. The speech synthesis method according to claim 5, characterized in that: The step of receiving the to-be-processed text by using a text encoder and converting the to-be-processed text into a code capable of representing the context and semantics of the text to obtain an encoding result includes: A text encoder using a Zipformer architecture receives the text to be processed and converts the text to be processed into a code that can represent the text context and semantics to obtain an encoding result; wherein the text encoder includes a U-Net structure, a convolution module, and an attention weight reuse mechanism.

7. The speech synthesis method according to claim 6, characterized in that: The method of using a stream matching decoder to predict the masked speech in combination with the encoding result, the interpolated noise, and the unmasked speech to obtain a prediction result includes: A stream matching decoder is used to combine the encoding result, the interpolated noise, and the unmasked speech to learn a time-dependent vector field, and the interpolated noise is converted into a target speech feature distribution by solving an ordinary differential equation to obtain a prediction result.

8. A speech synthesis device, characterized in that include: An acquisition unit, used to acquire the text to be processed, the interpolated noise, and the unmasked speech; A prediction unit is configured to input the text to be processed, the interpolated noise, and the unmasked speech into a synthesis model, wherein the synthesis model encodes the text to be processed, combines the adjusted noise and the unmasked speech, learns feature changes in the time dimension, and converts the noise into target speech features to form a prediction result; An output unit is used to output the prediction result.

9. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images