Non-regressive speech synthesis methods, devices, computer equipment, and storage media

Abstract

This application belongs to the field of speech synthesis technology, and relates to a non-autoregressive speech synthesis method, apparatus, computer equipment, and storage medium. The method includes: obtaining Mel spectrogram features and transcribed text based on audio prompt data; obtaining padded text based on padding characters and Mel spectrogram features; obtaining sequence length N based on padded text and transcribed text; obtaining concatenated text based on transcribed text and padded text; inputting the Mel spectrogram features and concatenated text into a trained DiT diffusion model to obtain diffusion noise data; inputting the diffusion noise data and sequence length N into an ordinary differential equation solver to obtain synthesized Mel spectrogram features; and performing an inverse Mel transform operation on the synthesized Mel spectrogram features using a vocoder to obtain the target synthesized audio. This application constructs a non-autoregressive model based on stream matching and a diffusion transformer, which can improve the quality and accuracy of generated speech.

Description

Technical Field

[0001] This application relates to the field of speech synthesis technology, applicable to the financial or medical fields, and particularly to non-autoregressive speech synthesis methods, devices, computer equipment, and storage media. Background Technology

[0002] In recent years, text-to-speech (TTS) technology has made significant progress. Currently, TTS models can generate speech for any text using only a few seconds of audio prompts, mimicking the speaker in the audio prompts. The generated speech has extremely high fidelity and naturalness, making it almost indistinguishable from human speech.

[0003] While autoregressive (AR)-based TTS models offer an intuitive approach to progressively predicting the next one or more tokens and have achieved encouraging results in zero-shot TTS capabilities, the inherent limitations of AR modeling still require additional work to address issues such as inference latency and exposure bias. Furthermore, the quality of the speech tokenizer is crucial for AR models to achieve high-fidelity synthesis.

[0004] Non-autoregressive models achieve fast inference through parallel processing and strike a good balance between synthesis quality and latency. Although autoregressive models excel in zero-shot performance, non-autoregressive models, especially diffusion-based models (such as FM-OT), show greater potential in TTS and other generative tasks.

[0005] However, the applicant found that achieving more flexible and natural alignment between text and speech remains a major challenge for non-autoregressive models. Summary of the Invention

[0006] The purpose of this application is to propose a non-autoregressive speech synthesis method, apparatus, computer device, and storage medium to solve the problem that non-autoregressive models cannot achieve more flexible and natural alignment between text and speech.

[0007] To address the aforementioned technical problems, this application provides a non-autoregressive speech synthesis method, employing the following technical solution:

[0008] Receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least audio prompt data and text prompt data:

[0009] Based on the audio prompt data, obtain the Mel spectrogram feature x. ref and transcribed text y ref ;

[0010] Based on the filler and the Mel spectrogram feature x refPerform character padding on the text prompt data to obtain the padded text y. gen ;

[0011] According to the text completion y gen and the transcribed text y ref Get the sequence length N;

[0012] For the transcribed text y ref And the text completion y gen Perform the concatenation operation to obtain the concatenated text z. ref·gen ;

[0013] The Mel spectrogram feature x ref The spliced ​​text z ref·gen The sequence length N is input into the trained DiT diffusion model for diffusion operation to obtain diffused noise data;

[0014] The diffused noise data is input into an ordinary differential equation solver for solution, resulting in the synthesized Mel spectrum features.

[0015] Based on the vocoder's analysis of the synthesized Mel spectrogram features Perform an inverse Mel transform operation to obtain the target synthesized audio.

[0016] Furthermore, the step of completing the text y... gen and the transcribed text y ref The steps to obtain the sequence length N specifically include the following steps:

[0017] According to the text completion y gen and the transcribed text y ref The sequence length N is obtained by performing a sequence length estimation operation based on the proportion of the number of characters.

[0018] Furthermore, the step of completing the text y... gen and the transcribed text y ref The steps to obtain the sequence length N specifically include the following steps:

[0019] Call the trained sequence length prediction model and input the padded text y gen and the transcribed text y ref The sequence length is input into the sequence length prediction model to perform a sequence length prediction operation, and the sequence length N is obtained.

[0020] Furthermore, in the transcribed text y ref And the text completion y gen Perform the concatenation operation to obtain the concatenated text z. ref·genBefore the steps described above, the following steps are also included:

[0021] The text to be filled in y gen The input is fed into a ConvNeXtV2 convolutional neural network for feature enhancement operations to obtain enhanced text;

[0022] The transcribed text y ref And the text completion y gen Perform the concatenation operation to obtain the concatenated text z. ref·gen The steps specifically include the following:

[0023] For the transcribed text y ref The enhanced text is then concatenated to obtain the concatenated text z. ref·gen .

[0024] Furthermore, in the process of defining the Mel spectrogram feature x ref The spliced ​​text z ref·gen After the step of inputting the sequence length N into the trained DiT diffusion model for diffusion operation to obtain diffused noise data, the following steps are also included:

[0025] The diffused noise data is subjected to a swing sampling operation based on the swing function S(t) to obtain swing sampled noise data;

[0026] The process involves inputting the diffused noise data and the sequence length N into an ordinary differential equation solver for solution, thereby obtaining the synthesized Mel spectrum features. The steps specifically include the following:

[0027] The swing sampling noise data and the sequence length N are input into an ordinary differential equation solver for solution operation to obtain the synthetic Mel spectrum features.

[0028] Furthermore, the swing function S(t) is expressed as:

[0029]

[0030] Where μ represents the peak position, σ represents the width of the control peak, and Z represents the normalization constant to ensure that S(t) is an effective probability distribution.

[0031] To address the aforementioned technical problems, this application also provides a non-regressive speech synthesis device, which employs the following technical solution:

[0032] The request receiving module is used to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least audio prompt data and text prompt data;

[0033] The audio processing module is used to obtain Mel spectrogram features x based on the audio cues data. ref and transcribed text y ref ;

[0034] The character completion module is used to fill in the blanks based on the filler character and the Mel spectrogram feature x. ref Perform character padding on the text prompt data to obtain the padded text y. gen ;

[0035] The sequence length acquisition module is used to obtain the length of the padded text y based on the sequence length. gen and the transcribed text y ref Get the sequence length N;

[0036] The splicing module is used to process the transcribed text y ref And the text completion y gen Perform the concatenation operation to obtain the concatenated text z. ref·gen ;

[0037] The diffusion module is used to diffuse the Mel spectrogram features x ref and the spliced ​​text z ref·gen The data is input into a trained DiT diffusion model to perform a diffusion operation, resulting in diffused noise data.

[0038] The ordinary differential equation solving module is used to input the diffused noise data and the sequence length N into the ordinary differential equation solver for solving the equation and obtaining the synthesized Mel spectrum features.

[0039] The inverse Mel transform module is used to analyze the features of the synthesized Mel spectrogram based on the vocoder. Perform an inverse Mel transform operation to obtain the target synthesized audio.

[0040] Furthermore, the sequence length acquisition module includes:

[0041] The length estimation submodule is used to estimate the length of the text y based on the length of the text. gen and the transcribed text y ref The sequence length N is obtained by performing a sequence length estimation operation based on the proportion of the number of characters.

[0042] To address the aforementioned technical problems, this application also provides a computer device that employs the following technical solution:

[0043] It includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the non-autoregressive speech synthesis method described above.

[0044] To address the aforementioned technical problems, this application also provides a computer-readable storage medium, employing the technical solution described below:

[0045] The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the non-autoregressive speech synthesis method described above.

[0046] This application provides a non-autoregressive speech synthesis method, comprising: receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least audio prompt data and text prompt data; and obtaining Mel spectrogram features x based on the audio prompt data. ref and transcribed text y ref Based on the filler and the Mel spectrogram feature x ref Perform character padding on the text prompt data to obtain the padded text ygen; based on the padded text ygen... gen and the transcribed text y ref Obtain the sequence length N; for the transcribed text y ref And the text completion y gen Perform the concatenation operation to obtain the concatenated text z. ref·gen ; the Mel spectrogram feature x ref and the spliced ​​text z ref·gen The data is input into a trained DiT diffusion model for diffusion operations to obtain diffusion noise data. The diffusion noise data and the sequence length N are then input into an ordinary differential equation solver for solving the equations to obtain the synthesized Mel spectrum features. Based on the vocoder's analysis of the synthesized Mel spectrogram features The inverse Mel transform operation is performed to obtain the target synthesized audio. Compared with existing technologies, this application constructs a non-autoregressive model based on stream matching and diffusion transform. Through a concise process, advanced model architecture, and efficient sampling strategy, it can more effectively utilize priming information, thereby improving the quality and accuracy of generated speech. Attached Figure Description

[0047] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is an exemplary system architecture diagram to which this application can be applied;

[0049] Figure 2This is a flowchart illustrating the implementation of the non-autoregressive speech synthesis method provided in the embodiments of this application;

[0050] Figure 3 This is a schematic diagram of the structure of the speech synthesis model provided in the embodiments of this application;

[0051] Figure 4 This is a schematic diagram of the structure of the non-regressive speech synthesis device provided in the embodiments of this application;

[0052] Figure 5 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0054] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0055] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0056] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0057] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0058] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0059] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0060] It should be noted that the non-autoregressive speech synthesis method provided in the embodiments of this application is generally executed by a server / terminal device, and correspondingly, the non-autoregressive speech synthesis device is generally set in the server / terminal device.

[0061] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0062] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of the non-autoregressive speech synthesis method according to this application. The non-autoregressive speech synthesis method includes steps S201, S202, S203, S204, S205, S206, and S207.

[0063] In step S201, a speech synthesis request sent by a user terminal is received, wherein the speech synthesis request includes at least audio prompt data and text prompt data.

[0064] In this embodiment, the user inputs a speech synthesis request through their terminal device (such as a mobile phone, computer, etc.), and this input is received by the system. The audio and text prompt data represent the specific content the user wishes the system to perform speech synthesis on. Specifically, the audio and text prompt data can be "transaction data, payment data, business data, or purchase data" related to financial institutions (such as banks). The audio and text prompt data can also be medical data related to medical scenarios, such as personal health records, prescriptions, and examination reports. It should be understood that the examples of audio and text prompt data provided here are for convenience of understanding only and are not intended to limit this application.

[0065] In step S202, the Mel spectrogram feature x is obtained based on the audio cues data. ref and transcribed text y ref .

[0066] In step S203, based on the filler symbol and the Mel spectrogram feature x ref Perform character padding on the text prompt data to obtain the padded text y. gen .

[0067] In step S204, based on the completed text y gen and transcribed text y ref Get the sequence length N.

[0068] In step S205, the transcribed text y ref And complete the text y gen Perform the concatenation operation to obtain the concatenated text z. ref·gen .

[0069] In step S206, the Mel spectrogram feature x ref , splicing text z ref·gen The sequence length N is input into the trained DiT diffusion model to perform diffusion operations, resulting in diffused noise data.

[0070] In step S207, the diffused noise data is input into the ordinary differential equation solver for solution operation to obtain the synthetic Mel spectrum characteristics.

[0071] In step S208, based on the vocoder's analysis of the synthesized Mel spectrogram features... Perform an inverse Mel transform operation to obtain the target synthesized audio.

[0072] In the embodiments of this application, the user terminal refers to a terminal device used to execute the image processing method for preventing document abuse provided in this application. The user terminal may be a mobile terminal such as a mobile phone, smartphone, laptop, digital broadcast receiver, PDA (personal digital assistant), PAD (tablet computer), PMP (portable multimedia player), navigation device, etc., as well as a fixed terminal such as a digital TV, desktop computer, etc. It should be understood that the examples of user terminals here are only for convenience of understanding and are not intended to limit this application.

[0073] In the embodiments of this application, see Figure 3 The schematic diagram of the speech synthesis model shown provides a fully non-autoregressive speech synthesis model based on Stream Matching and Diffusion Transformer (DiT). Unlike traditional methods, this approach does not require complex designs such as duration models, text encoders, or phoneme alignment. It simply pads the text input with padding characters to make its length the same as the input speech, and then generates speech through a denoising process. This method was initially proven feasible by E2TTS. However, the original design of E2TTS suffered from slow convergence and low robustness, making it difficult to widely apply.

[0074] In this application embodiment, an advanced text-to-speech (TTS) synthesis system is constructed. The model is trained based on a text-guided speech completion task. During training, no phoneme-level duration predictor is required, and higher naturalness is achieved in zero-shot generation while avoiding explicit phoneme alignment. This application adopts a similar process to E2TTS and proposes an improved architecture aimed at addressing two main problems in E2TTS: slow convergence speed (good timbre learning in the early stages of training, but difficulty in alignment learning) and insufficient robustness (difficulty in handling complex generation scenarios).

[0075] In this embodiment, the goal of the speech filling task is to predict a speech segment based on surrounding audio and complete text (including the surrounding text and the text to be generated). For simplicity, this application uses the symbol x to represent an audio sample and y to represent the corresponding text transcription, forming a data pair (x, y). Figure 1 As shown, the acoustic input during training is the Mel spectrogram features extracted from the audio sample x. Where F represents the dimension of the Mel spectrogram and N represents the sequence length.

[0076] Within the framework of CFM (Contextual Flow Matching), this application inputs noisy speech (1-t)x into the model. o +tx1 and the masked speech (1-m)⊙x1, where x orepresents Gaussian noise obtained by sampling, t represents the sampling flow step, m∈{0,1} F×N is a binary time-series mask.

[0077] According to the method of E2TTS, the present application directly uses English letters and symbols. For Chinese, the present application chooses to use complete pinyin to facilitate zero-shot generation. By decomposing the original text into a character sequence and padding it with padding symbols to the same length as the number of frames of the mel-spectrogram, the present application forms an extended sequence z, where C i represents the i-th character.

[0078] 1. Noisy speech:

[0079] Noisy Speech=(1-t)x o +tε

[0080] where, x o is the sampled Gaussian noise, and t is the sampling flow step.

[0081] 2. Masked speech:

[0082] Masked Speech=(1-m)⊙x₁

[0083] where, m∈{0, 1} F×N is a binary time-series mask.

[0084] 3. Extended character sequence:

[0085]

[0086] where, L z is the length of the character sequence, and L z =N.

[0087] During the training process, the present application constructs a complete training process through noisy and masked speech inputs and the extended character sequence. This method not only simplifies the model design, but also provides an effective training signal for subsequent speech generation tasks. Through this process, the model can better learn the mapping relationship between text and speech and achieve high-quality speech synthesis.

[0088] In the embodiment of the present application, in order to generate speech with required content, the present application uses the mel-spectrogram feature x of the audio prompt ref , its transcription text y ref and text prompt y gen . The audio prompt is used to provide speaker features, and the text prompt is used to guide the content of the generated speech.

[0089] In order to sample from the learned distribution, the converted mel feature xref , and the concatenated and expanded character sequence z ref·gen As a condition, this application includes:

[0090] 1. Conditional input:

[0091] Condition = (x ref , z ref·gen )

[0092] Where, x ref It is the Mel spectrogram feature of the audio cues, z ref·gen It is a sequence of characters that has been concatenated and expanded.

[0093] 2. Sampling process:

[0094] The process of sampling from the learned distribution to generate speech can be represented by the following formula:

[0095] x gen ~P(x|x ref , z ref·gen )

[0096] Where, x gen It is the Mel spectrogram of the generated speech.

[0097] 3. Post-processing:

[0098] To convert the generated Mel spectrogram into an audio signal, the Inverse Mel Transform can be used:

[0099] Audio = InverseMel(x gen )

[0100] This step converts the generated Mel spectrogram into an audible audio signal.

[0101] During the inference phase, this application generates speech with the desired content and speaker features by using a combination of audio and text prompts. Through simple duration estimation and conditional sampling, this application achieves high-quality speech generation while maintaining model simplicity. The generated Mel spectrogram is converted into an audio signal through inverse Mel transform, thus completing the speech generation process.

[0102] from Figure 3 As can be seen, this application starts with a sampled noise x0 and aims to obtain the other end of the flow process x1. Therefore, this application uses an ordinary differential equation (ODE) solver, integrating stepwise from ψ0(x0) = x0 to ψ1(x0) = x1, with the derivative being... During inference, the flow steps are provided in an ordered manner, for example, a certain number of steps are uniformly sampled from 0 to 1 according to the NFE (Number of Function Evaluations) setting.

[0103] In this embodiment of the application, after obtaining model v t After obtaining the Mel spectrum from the 0DE solver, this application discards x. ref Partially. Then, this application uses a vocoder to convert the Mel spectrogram back into a speech signal.

[0104] In this embodiment, the inference process begins with sampled noise and gradually generates the target Mel spectrogram x1 using an ODE solver. During generation, this application provides sequential streaming steps according to the NFE settings. After generation, this application discards the audio cues and uses a vocoder to convert the Mel spectrogram into the final speech signal.

[0105] In this embodiment, the CFM flow step t is used as a condition for adaLN-zero, rather than being appended to the concatenated input sequence as in Voicebox. This application finds that the additional mean pooling of the text sequence to the tokens is not necessary for the adaLN condition, possibly because TTS tasks require more rigorously guided results, while mean pooling of text tokens is too coarse.

[0106] In some optional implementations of the embodiments of this application, this application employs some positional embedding settings from Voicebox. The flow step is encoded using sinusoidal positional embedding, and the concatenated input sequence is then augmented with convolutional positional embedding. This application applies Rotational Positional Embedding (RoPE) to self-attention instead of symmetric bidirectional ALiBi bias. For the expanded character sequence, this application also adds absolute sinusoidal positional embedding before inputting it into the Con v NeXt module.

[0107] In this embodiment of the application, a non-autoregressive speech synthesis method is provided, comprising: receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least audio prompt data and text prompt data; and obtaining Mel spectrogram features x based on the audio prompt data. ref and transcribed text y ref Based on the filler and Mel spectrogram features x ref Perform character padding on the text prompt data to obtain the padded text y. gen ; Based on the completed text y gen and transcribed text y ref Obtain the sequence length N; for the transcribed text y ref And complete the text y genPerform the concatenation operation to obtain the concatenated text z. ref·gen ; to use Mel spectrogram features x ref and splicing text z ref·gen The data is input into a trained DiT diffusion model for diffusion operations to obtain diffusion noise data. The diffusion noise data and sequence length N are then input into an ordinary differential equation solver for solving the equations, yielding the synthetic Mel-ray spectrogram features. Based on the characteristics of the synthesized Mel spectrogram by the vocoder The inverse Mel transform operation is performed to obtain the target synthesized audio. Compared with existing technologies, this application constructs a non-autoregressive model based on stream matching and diffusion transform. Through a concise process, advanced model architecture, and efficient sampling strategy, it can more effectively utilize priming information, thereby improving the quality and accuracy of generated speech.

[0108] In some optional implementations of the embodiments of this application, the above is based on the text completion y gen and transcribed text y ref The steps to obtain the sequence length N specifically include the following steps:

[0109] According to the completed text y gen and transcribed text y ref The sequence length N is obtained by estimating the sequence length based on the proportion of characters.

[0110] In some optional implementations of the embodiments of this application, the above is based on the text completion y gen and transcribed text y ref The steps to obtain the sequence length N specifically include the following steps:

[0111] Call the trained sequence length prediction model and pad the text y gen and transcribed text y ref The sequence length is input into the sequence length prediction model to perform sequence length prediction and obtain the sequence length N.

[0112] In this embodiment, the sequence length N (i.e., the length) now becomes a key factor, needing to inform the model of the required length for generating samples. This application can train a single model based on x. ref y ref and y gen To predict and provide the duration. Here, this application simply uses y gen and y ref The duration is estimated by the proportion of characters in the spectrum. This application assumes that the total length of the characters does not exceed the length of the Mel spectrogram, and therefore padding is performed using padding characters as during training.

[0113] In some optional implementations of the embodiments of this application, when transcribing text y...ref And complete the text y gen Perform the concatenation operation to obtain the concatenated text z. ref·gen Before the steps described above, the following steps are also included:

[0114] Will complete the text y gen The input is fed into a ConvNeXtV2 convolutional neural network for feature enhancement operations to obtain enhanced text;

[0115] For transcribed text y ref And complete the text y gen Perform the concatenation operation to obtain the concatenated text z. ref·gen The steps specifically include the following:

[0116] For transcribed text y ref And perform concatenation operations on the enhanced text to obtain the concatenated text z. ref·gen .

[0117] In this application embodiment, in order to enhance the text alignment capability of the model, the Co nvNeXtV2 module is also introduced.

[0118] In this embodiment, the model input includes character sequences, noisy speech, and masked speech. Before feature dimension concatenation, the character sequences are processed separately using the ConvNeXt module. Experiments show that this approach provides an independent modeling space for the text input, allowing it to better prepare itself for subsequent contextual learning. Unlike the forced phoneme-level alignment in Voicebox, this application does not explicitly introduce strict boundaries for the text, but instead allows semantic and acoustic features to be learned jointly throughout the model. Furthermore, unlike E2TTS which uses inputs with significantly different lengths (different effective information lengths), this application's design mitigates this length difference.

[0119] In some optional implementations of the embodiments of this application, when the Mel spectrogram feature x ref , splicing text z ref·gen After the step of inputting the sequence length N into the trained DiT diffusion model for diffusion operation to obtain diffused noise data, the following steps are also included:

[0120] The diffused noise data is obtained by performing a swing sampling operation on the swing function S(t);

[0121] The diffused noise data and sequence length N are input into the ordinary differential equation solver for solving, and the synthesized Mel spectrum features are obtained. The steps specifically include the following:

[0122] The swing sampling noise data and the sequence length N are input into the ordinary differential equation solver for solution operation, and the synthesized Mel spectrum features are obtained.

[0123] In this application, a "swing sampling" strategy is proposed for inference, which significantly improves the performance and efficiency of the model. This sampling strategy can be easily applied to existing flow-matching-based models without retraining. The design of this application not only accelerates training speed but also significantly improves inference speed compared to existing diffusion-based TTS models.

[0124] In this embodiment, CFM can be viewed as a model that uses cosine scheduling for v prediction. In image synthesis, a unimodal logit-normal sampling strategy is proposed to further schedule flow steps, allowing for more frequent sampling of intermediate flow steps and thus assigning them greater weights. This application hypothesizes that this sampling method can make the learning difficulty of the model more evenly distributed across different flow steps t∈[0,1].

[0125] In contrast, the model in this application uses traditional uniform sampling t ~ U[0,1] during training, but applies non-uniform sampling during inference. Specifically, this application defines a "swing sampling" function:

[0126] Swing sampling function

[0127] 1. Definition:

[0128] The swing sampling function S(t) is used to generate flow steps t during inference. This function dynamically adjusts the sampling density according to the stage at which samples are generated.

[0129] This application defines S(t) as follows:

[0130]

[0131] Where μ is the peak position, σ controls the width of the peak, and Z is a normalization constant to ensure that S(t) is an effective probability distribution.

[0132] 2. Sampling process:

[0133] During the inference process, this application uses S(t) to sample the flow step t. The specific steps are as follows:

[0134] First, sample a flow step t according to S(t).

[0135] Then, use t to perform the inference steps of the model.

[0136] This approach allows the application to perform non-uniform sampling of different flow steps during inference, thereby making more efficient use of the model's generative capabilities.

[0137] 3. Parameter adjustment:

[0138] The values ​​of μ and σ can be adjusted according to specific task requirements. For example, in stages where more detail generation is needed, μ can be set to 0.5, and σ can be reduced to focus sampling in intermediate steps.

[0139] In this embodiment, by employing a swing sampling strategy, the model can more flexibly control the sampling distribution of flow steps during inference, thereby improving efficiency while maintaining generation quality. This method is not only applicable to the model of this application, but can also be easily applied to other CFM-based models to further improve their generation performance and efficiency.

[0140] In this embodiment, the CFM model focuses on outlining the speech from pure noise in the early stages (when t→0), while in the later stages it focuses more on refining details. Therefore, the alignment of speech and text is determined based on the first few generated results. By introducing a scaling parameter S<0, this application allows the model to perform more inference when the t value is small, thereby providing more startup information for the ODE solver to perform more accurate evaluation in the initial integration step.

[0141] In this embodiment, this strategy enables the model to better capture the overall contour of speech in the early stages of speech generation, while focusing on refining details in later stages. By adjusting the value of t, the model can more effectively utilize priming information, thereby improving the quality and accuracy of the generated speech.

[0142] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0143] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0144] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0145] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0146] Further reference Figure 4 As a response to the above Figure 2 The implementation of the method shown in this application provides an embodiment of a non-autoregressive speech synthesis device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0147] like Figure 4 As shown, the non-regressive speech synthesis device 200 of this application embodiment includes:

[0148] The request receiving module 210 is used to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least audio prompt data and text prompt data;

[0149] Audio processing module 220 is used to obtain Mel spectrogram features x based on audio cues data. ref and transcribed text y ref ;

[0150] Character completion module 230 is used to fill in characters based on padding characters and Mel spectrogram features x. ref Perform character padding on the text prompt data to obtain the padded text y. gen ;

[0151] The sequence length acquisition module 240 is used to obtain the length of the padded text y based on the sequence length. genand transcribed text y ref Get the sequence length N;

[0152] The splicing module 250 is used for processing transcribed text y ref And complete the text y gen Perform the concatenation operation to obtain the concatenated text z. ref·gen ;

[0153] Diffusion module 260 is used to diffuse the Mel spectrogram features x ref and splicing text z ref·gen The data is input into a trained DiT diffusion model to perform a diffusion operation, resulting in diffused noise data.

[0154] The ordinary differential equation solving module 270 is used to input the diffused noise data and the sequence length N into the ordinary differential equation solver for solving the problem and obtaining the synthetic Mel spectrum features.

[0155] Inverse Mel transform module 280, used to transform the synthesized Mel spectrogram features based on the vocoder. Perform an inverse Mel transform operation to obtain the target synthesized audio.

[0156] In this embodiment of the application, a non-autoregressive speech synthesis apparatus 200 is provided, comprising: a request receiving module 210, configured to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least audio prompt data and text prompt data; and an audio processing module 220, configured to obtain Mel spectrogram features x based on the audio prompt data. ref and transcribed text y ref Character completion module 230, used to fill in characters based on padding characters and Mel spectrogram features x. ref Perform character padding on the text prompt data to obtain the padded text y. gen ; Sequence length acquisition module 240, used to obtain the length of the padded text y gen and transcribed text y ref Obtain the sequence length N; splicing module 250, used for processing the transcribed text y ref And complete the text y gen Perform the concatenation operation to obtain the concatenated text z. ref·gen ;Diffusion module 260, used to diffuse the Mel spectrogram features x ref and splicing text z ref·gen The data is input into a trained DiT diffusion model for diffusion operations to obtain diffusion noise data. The ordinary differential equation solving module 270 is used to input the diffusion noise data and the sequence length N into the ordinary differential equation solver for solving operations to obtain the synthetic Mel spectrum features. Inverse Mel transform module 280, used to transform the synthesized Mel spectrogram features based on the vocoder. The inverse Mel transform operation is performed to obtain the target synthesized audio. Compared with existing technologies, this application constructs a non-autoregressive model based on stream matching and diffusion transform. Through a concise process, advanced model architecture, and efficient sampling strategy, it can more effectively utilize priming information, thereby improving the quality and accuracy of generated speech.

[0157] In some optional implementations of the embodiments of this application, the sequence length acquisition module includes:

[0158] The length estimation submodule is used to estimate the length of the text based on the padding text y. gen and transcribed text y ref The sequence length N is obtained by estimating the sequence length based on the proportion of characters.

[0159] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference] for details. Figure 5 , Figure 5 This is a basic structural block diagram of a computer device according to an embodiment of this application.

[0160] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected via a system bus. It should be noted that only the computer device 300 with components 310-330 is shown in the figure; however, it should be understood that it is not required to implement all the shown components, and more or fewer components can be implemented alternatively. Those skilled in the art will understand that the computer device described here is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0161] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0162] The memory 310 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 may be an internal storage unit of the computer device 300, such as the hard disk or memory of the computer device 300. In other embodiments, the memory 310 may also be an external storage device of the computer device 300, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 300. Of course, the memory 310 may also include both internal storage units and external storage devices of the computer device 300. In this embodiment, the memory 310 is typically used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions for non-autoregressive speech synthesis methods. Furthermore, the memory 310 can also be used to temporarily store various types of data that have been output or will be output.

[0163] In some embodiments, the processor 320 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 320 is typically used to control the overall operation of the computer device 300. In this embodiment, the processor 320 is used to execute computer-readable instructions stored in the memory 310 or to process data, for example, to execute computer-readable instructions for the non-autoregressive speech synthesis method.

[0164] The network interface 330 may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 300 and other electronic devices.

[0165] The computer device provided in this application constructs a non-autoregressive model based on stream matching and diffusion transformers. Through a concise process, advanced model architecture, and efficient sampling strategy, it can more effectively utilize priming information, thereby improving the quality and accuracy of generated speech.

[0166] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by at least one processor to cause the at least one processor to perform the steps of the non-autoregressive speech synthesis method described above.

[0167] The computer-readable storage medium provided in this application constructs a non-autoregressive model based on stream matching and diffusion transformers. Through a concise process, advanced model architecture, and efficient sampling strategy, it can more effectively utilize priming information, thereby improving the quality and accuracy of generated speech.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0169] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to provide a more thorough and comprehensive understanding of the disclosure of this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application.

Claims

1. A non-autoregressive speech synthesis method, characterized in that, Includes the following steps: The system receives a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least audio prompt data and text prompt data, and the audio prompt data and the text prompt data are the specific content that the user wants the system to perform speech synthesis. Obtain Mel spectrogram features based on the audio prompt data. and transcribed text ; Based on the filler and the Mel spectrogram features Perform character padding on the text prompt data to obtain the padded text. ; According to the text completion and the transcribed text Get sequence length N ; For the transcribed text and the completed text Perform the concatenation operation to obtain the concatenated text. ; The Mel spectrogram features The spliced ​​text and the sequence length N The data is input into a trained DiT diffusion model to perform a diffusion operation, resulting in diffused noise data. The diffused noise data is input into an ordinary differential equation solver for solution, resulting in the synthesized Mel spectrum features. ; Based on the vocoder's analysis of the synthesized Mel spectrogram features Perform an inverse Mel transform operation to obtain the target synthesized audio; The DiT diffusion model employs traditional uniform sampling t ~ U[0,1] during training, and non-uniform sampling during inference. The inference process generates speech with the desired content and speaker features using a combination of audio and text prompts. During inference, a swing function is used... A flow step t is sampled, and the inference steps of the DiT diffusion model after swing sampling are performed based on the flow step t, wherein the swing function is... Represented as: in, Indicates the peak position. Indicates the width of the control peak. Normalization constant, ensuring It is an effective probability distribution.

2. The non-autoregressive speech synthesis method according to claim 1, characterized in that, The text completion method and the transcribed text Get sequence length N The steps specifically include the following: According to the text completion and the transcribed text The sequence length is estimated by performing a sequence length estimation operation based on the proportion of the number of characters. N .

3. The non-autoregressive speech synthesis method according to claim 1, characterized in that, The text completion method and the transcribed text Get sequence length N The steps specifically include the following: Call the trained sequence length prediction model and add the padded text. and the transcribed text The sequence length is input into the sequence length prediction model to perform a sequence length prediction operation, thereby obtaining the sequence length. N .

4. The non-autoregressive speech synthesis method according to claim 1, characterized in that, In the transcribed text and the completed text Perform the concatenation operation to obtain the concatenated text. Before the steps described above, the following steps are also included: The text to be filled in The input is fed into a ConvNeXtV2 convolutional neural network for feature enhancement operations to obtain enhanced text; The transcribed text and the completed text Perform the concatenation operation to obtain the concatenated text. The steps specifically include the following: For the transcribed text The enhanced text is then concatenated to obtain the concatenated text. .

5. A non-autoregressive speech synthesis device, characterized in that, include: The request receiving module is used to receive a speech synthesis request sent by a user terminal. The speech synthesis request includes at least audio prompt data and text prompt data, wherein the audio prompt data and the text prompt data are the specific content that the user wants the system to perform speech synthesis. The audio processing module is used to obtain Mel spectrogram features based on the audio cues data. and transcribed text ; The character completion module is used to fill in the characters based on the padding characters and the Mel spectrogram features. Perform character padding on the text prompt data to obtain the padded text. ; The sequence length acquisition module is used to obtain the length of the padded text. and the transcribed text Get sequence length N ; The splicing module is used to process the transcribed text. and the completed text Perform the concatenation operation to obtain the concatenated text. ; A diffusion module is used to diffuse the Mel spectrogram features. and the spliced ​​text The data is input into a trained DiT diffusion model to perform a diffusion operation, resulting in diffused noise data. The ordinary differential equation solving module is used to solve the diffused noise data and the sequence length. N The input is fed into an ordinary differential equation solver for solution operation, and the synthesized Mel spectrum features are obtained. ; The inverse Mel transform module is used to analyze the features of the synthesized Mel spectrogram based on the vocoder. Perform an inverse Mel transform operation to obtain the target synthesized audio; The DiT diffusion model employs traditional uniform sampling t ~ U[0,1] during training, and non-uniform sampling during inference. The inference process generates speech with the desired content and speaker features using a combination of audio and text prompts. During inference, a swing function is used... A flow step t is sampled, and the inference steps of the DiT diffusion model after swing sampling are performed based on the flow step t, wherein the swing function is... Represented as: in, Indicates the peak position. Indicates the width of the control peak. Normalization constant, ensuring It is an effective probability distribution.

6. The non-regressive speech synthesis device according to claim 5, characterized in that, The sequence length acquisition module includes: The length estimation submodule is used to estimate the length of the text based on the length of the text. and the transcribed text The sequence length is estimated by performing a sequence length estimation operation based on the proportion of the number of characters. N .

7. A computer device, comprising a memory and a processor, characterized in that, The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the non-autoregressive speech synthesis method as described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the non-autoregressive speech synthesis method as described in any one of claims 1 to 4.

Images