Non-autoregressive speech synthesis method and device, computer equipment and storage medium

Abstract

The invention belongs to the technical field of speech synthesis, and relates to a non-autoregressive speech synthesis method and device, computer equipment and a storage medium, and the method comprises the steps: obtaining a Mel spectrogram feature and a transcriptional text according to audio prompt data; obtaining a supplementary text of the text prompt data according to the filler and the Mel spectrogram features; obtaining a sequence length N according to the complementary text and the transcriptional text; obtaining a spliced text according to the transcriptional text and the complementary text; inputting the Mel spectrogram features and the spliced text into a trained Di T diffusion model to obtain diffusion noise data; inputting the diffusion noise data and the sequence length N into an ordinary differential equation solver to obtain synthetic Mel spectrogram features; and performing inverse Mel transform operation on the synthesized Mel spectrogram features according to a vocoder to obtain a target synthesized audio. According to the invention, a non-autoregressive model is constructed based on stream matching and a diffusion converter, so that the quality and accuracy of voice generation can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, applicable to the financial or medical fields, and in particular to a non-autoregressive speech synthesis method, apparatus, computer equipment and storage medium. Background Art

[0002] Text-to-speech (TTS) technology has made significant progress in recent years. Current TTS models can generate speech for arbitrary text given only a few seconds of audio prompts and mimic the speaker in the audio prompt. The generated speech has extremely high fidelity and naturalness, and is almost indistinguishable from human speech.

[0003] Although autoregression (AR)-based TTS models provide an intuitive way to gradually predict the next token or tokens and have achieved encouraging results in zero-shot TTS capabilities, the inherent limitations of AR modeling still require additional work to address issues such as inference latency and exposure bias. In addition, the quality of the speech tagger is critical for AR models to achieve high-fidelity synthesis.

[0004] Non-autoregressive models achieve fast inference through parallel processing and strike a good balance between synthesis quality and latency. Although autoregressive models excel in zero-shot performance, non-autoregressive models, especially diffusion-based models such as FM-OT, show greater potential in TTS and other generation tasks.

[0005] However, the applicants found that how to achieve a more flexible and natural alignment between text and speech is still the main challenge facing non-autoregressive models. Summary of the invention

[0006] The purpose of the embodiments of the present application is to propose a non-autoregressive speech synthesis method, apparatus, computer equipment and storage medium to solve the problem that the non-autoregressive model cannot achieve a more flexible and natural alignment between text and speech.

[0007] In order to solve the above technical problems, the embodiment of the present application provides a non-autoregressive speech synthesis method, which adopts the following technical solution:

[0008] Receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least audio prompt data and text prompt data:

[0009] Obtain the Mel-spectrogram feature x according to the audio prompt data ref and the transcript ref ;

[0010] According to the filler and the Mel-spectrogram feature x refPerform character completion operation on the text prompt data to obtain the completed text y gen ;

[0011] Complete the text y according to gen and the transcribed text y ref Get the sequence length N;

[0012] The transcribed text y ref And the completed text y gen Perform the splicing operation to obtain the spliced ​​text z ref·gen ;

[0013] The Mel-spectrogram feature x ref , the concatenated text z ref·gen The sequence length N is input into the trained DiT diffusion model to perform diffusion operation to obtain diffusion noise data;

[0014] The diffuse noise data is input into the ordinary differential equation solver for solving, and the synthesized Mel spectrum feature is obtained.

[0015] According to the vocoder, the synthesized Mel-spectrogram features Perform an inverse Mel transform operation to obtain the target synthesized audio.

[0016] Further, the completion text y gen and the transcribed text y ref The step of obtaining the sequence length N specifically includes the following steps:

[0017] Complete the text y according to gen and the transcribed text y ref The sequence length is estimated based on the ratio of the number of characters to obtain the sequence length N.

[0018] Further, the completion text y gen and the transcribed text y ref The step of obtaining the sequence length N specifically includes the following steps:

[0019] Call the trained sequence length prediction model and fill in the text y gen and the transcribed text y ref The sequence length is input into the sequence length prediction model to perform a sequence length prediction operation to obtain the sequence length N.

[0020] Further, in the transcription text y ref And the completed text y gen Perform the splicing operation to obtain the spliced ​​text z ref·genBefore the step of:

[0021] The padded text y gen Input into ConvNeXtV2 convolutional neural network for feature enhancement operation to obtain enhanced text;

[0022] The transcribed text y ref And the completed text y gen Perform the splicing operation to obtain the spliced ​​text z ref·gen The steps specifically include the following steps:

[0023] The transcribed text y ref And the enhanced text is spliced ​​to obtain the spliced ​​text z ref·gen .

[0024] Further, in the step of converting the Mel frequency spectrum feature x ref , the concatenated text z ref·gen After the step of inputting the sequence length N into the trained DiT diffusion model for diffusion operation to obtain diffusion noise data, the following steps are also included:

[0025] Performing a swing sampling operation on the diffusion noise data according to the swing function S(t) to obtain swing sampling noise data;

[0026] The diffuse noise data and the sequence length N are input into the ordinary differential equation solver for solving the problem, and the synthesized Mel spectrum feature is obtained. The steps specifically include the following steps:

[0027] The swing sampling noise data and the sequence length N are input into the ordinary differential equation solver for solving, and the synthetic Mel spectrum feature is obtained.

[0028] Furthermore, the swing function S(t) is expressed as:

[0029]

[0030] Among them, μ represents the peak position, σ represents the width of the controlled peak, and Z represents the normalization constant to ensure that S(t) is a valid probability distribution.

[0031] In order to solve the above technical problems, the embodiment of the present application further provides a non-autoregressive speech synthesis device, which adopts the following technical solution:

[0032] A request receiving module, used to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least audio prompt data and text prompt data;

[0033] An audio processing module, used to obtain a Mel-spectrogram feature x according to the audio prompt data ref and the transcript ref ;

[0034] The character completion module is used to fill in the character according to the filler and the Mel spectrum feature x ref Perform character completion operation on the text prompt data to obtain the completed text y gen ;

[0035] A sequence length acquisition module is used to complete the text y according to the gen and the transcribed text y ref Get the sequence length N;

[0036] A splicing module is used to transcribe the text y ref And the completed text y gen Perform the splicing operation to obtain the spliced ​​text z ref·gen ;

[0037] Diffusion module, used to transform the Mel frequency spectrum feature x ref And the concatenated text z ref·gen Input into the trained DiT diffusion model for diffusion operation to obtain diffusion noise data;

[0038] An ordinary differential equation solving module is used to input the diffuse noise data and the sequence length N into the ordinary differential equation solver for solving the problem, and obtain the synthetic Mel spectrum feature.

[0039] An inverse Mel transform module is used to transform the synthesized Mel spectrogram features according to the vocoder Perform an inverse Mel transform operation to obtain the target synthesized audio.

[0040] Furthermore, the sequence length acquisition module includes:

[0041] The length estimation submodule is used to complete the text y according to the gen and the transcribed text y ref The sequence length is estimated based on the ratio of the number of characters to obtain the sequence length N.

[0042] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0043] The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the non-autoregressive speech synthesis method as described above when executing the computer-readable instructions.

[0044] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0045] The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the non-autoregressive speech synthesis method described above.

[0046] The present application provides a non-autoregressive speech synthesis method, comprising: receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least audio prompt data and text prompt data; obtaining a Mel-spectrogram feature x according to the audio prompt data ref and the transcript ref ; According to the filler and the Mel spectrum feature x ref Perform character completion operation on the text prompt data to obtain a completed text ygen; gen and the transcribed text y ref Get the sequence length N; for the transcription text y ref And the completed text y gen Perform the splicing operation to obtain the spliced ​​text z ref·gen ; The Mel spectrum feature x ref And the concatenated text z ref·gen Input the trained DiT diffusion model to perform diffusion operation to obtain diffusion noise data; input the diffusion noise data and the sequence length N to the ordinary differential equation solver for solution operation to obtain the synthetic Mel spectrum feature According to the vocoder, the synthesized Mel-spectrogram features An inverse Mel transform operation is performed to obtain the target synthesized audio. Compared with the prior art, this application constructs a non-autoregressive model based on stream matching and diffusion transformer, which can more effectively utilize the start information through a concise process, advanced model architecture and efficient sampling strategy, thereby improving the quality and accuracy of the generated speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0049] Figure 2is a flowchart of the implementation of the non-autoregressive speech synthesis method provided in the embodiment of the present application;

[0050] Figure 3 It is a structural diagram of a speech synthesis model provided in an embodiment of the present application;

[0051] Figure 4 is a structural schematic diagram of a non-autoregressive speech synthesis device provided in an embodiment of the present application;

[0052] Figure 5 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0053] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0054] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0055] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0056] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0057] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0058] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0059] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0060] It should be noted that the non-autoregressive speech synthesis method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the non-autoregressive speech synthesis device is generally arranged in the server / terminal device.

[0061] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0062] Continue to refer Figure 2 , shows a flow chart of an embodiment of a non-autoregressive speech synthesis method according to the present application. The non-autoregressive speech synthesis method comprises: step S201, step S202, step S203, step S204, step S205, step S206 and step S207.

[0063] In step S201, a speech synthesis request sent by a user terminal is received, wherein the speech synthesis request includes at least audio prompt data and text prompt data.

[0064] In an embodiment of the present application, the user inputs a speech synthesis request through their terminal device (such as a mobile phone, computer, etc.), and this input is received by the system. The audio prompt data and text prompt data are the specific content that the user wants the system to perform speech synthesis. Specifically, the audio prompt data and text prompt data can be "transaction data or payment data or business data or purchase data" related to financial institutions (such as banks, etc.). The audio prompt data and text prompt data can also be medical data related to medical scenarios, such as personal health records, prescriptions, examination reports and other data. It should be understood that the examples of audio prompt data and text prompt data here are only for the convenience of understanding and are not used to limit this application.

[0065] In step S202, the Mel-spectrogram feature x is obtained according to the audio prompt data. ref and the transcript ref .

[0066] In step S203, according to the filler and the Mel-spectrogram feature x ref Perform character completion operation on the text prompt data to obtain the completed text y gen .

[0067] In step S204, according to the completion text y gen and the transcript ref Get the sequence length N.

[0068] In step S205, the transcribed text y ref And complete the text y gen Perform the splicing operation to obtain the spliced ​​text z ref·gen .

[0069] In step S206, the Mel-spectrogram feature x ref , splice text z ref·gen And the sequence length N is input into the trained DiT diffusion model for diffusion operation to obtain diffusion noise data.

[0070] In step S207, the diffuse noise data is input into the ordinary differential equation solver for solving, and the synthesized Mel spectrum feature is obtained.

[0071] In step S208, the mel spectrogram feature is synthesized according to the vocoder Perform an inverse Mel transform operation to obtain the target synthesized audio.

[0072] In the embodiments of the present application, the user terminal refers to a terminal device used to execute the image processing method for preventing document abuse provided by the present application. The user terminal can be a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, etc., as well as a fixed terminal such as a digital TV, a desktop computer, etc. It should be understood that the examples of user terminals here are only for convenience of understanding and are not used to limit the present application.

[0073] In the embodiments of this application, see Figure 3 The structural diagram of the speech synthesis model shown in Figure 1 provides a completely non-autoregressive speech synthesis model based on stream matching and diffusion transformer (DiT). Unlike traditional methods, this scheme does not require complex designs such as duration models, text encoders, or phoneme alignment. Simply pad the text input with fillers to make it the same length as the input speech, and then generate speech through a denoising process. This method was originally proven to be feasible by E2TTS. However, the original design of E2TTS has problems with slow convergence and low robustness, making it difficult to be widely used.

[0074] In an embodiment of the present application, the present application constructs an advanced text-to-speech (TTS) synthesis system. The present application trains the model of the present application based on a text-guided speech completion task. No phoneme-level duration predictor is required during the training process, and higher naturalness can be achieved in zero-sample generation while avoiding explicit phoneme alignment. The present application adopts a process similar to E2TTS and proposes an improved architecture of the present application, which aims to solve the two main problems existing in E2TTS: slow convergence (better timbre learning in the early stages of training, but difficult alignment learning) and insufficient robustness (difficult to handle generation in complex situations).

[0075] In the embodiment of the present application, the goal of the speech completion task is to predict a speech segment based on the surrounding audio and the complete text (including the surrounding text and the text to be generated). To simplify the description, the present application uses the symbol x to represent the audio sample and y to represent the corresponding text transcription, forming a data pair (x, y). Figure 1 As shown, the acoustic input during training is the Mel-spectrogram feature extracted from the audio sample x Where F represents the dimension of the Mel-spectrogram and N represents the sequence length.

[0076] In the framework of CFM (Contextual Flow Matching), this application inputs the noisy speech (1-t)x into the model. o +tx1 and the masked speech (1-m)⊙x1, where x orepresents the sampled Gaussian noise, t represents the sampled flow step, m∈{0,1} F×N is a binary temporal mask.

[0077] According to the E2TTS method, this application directly uses English letters and symbols. For Chinese, this application chooses to use complete pinyin to facilitate zero-sample generation. By decomposing the original text into character sequences and padding them with fillers to the same length as the number of Mel spectrogram frames, this application forms an extended sequence z, where C i Represents the i-th character.

[0078] 1. Noisy speech:

[0079] Noisy Speech = (1-t) x o +tx1

[0080] Among them, x o is the sampled Gaussian noise and t is the sampling flow step.

[0081] 2. Masked speech:

[0082] Masked Speech=(1-m)⊙x1

[0083] Where m∈{0,1} F×N is a binary temporal mask.

[0084] 3. Extended character sequence:

[0085]

[0086] Among them, L z is the length of the character sequence, and L z =N.

[0087] During the training process, this application constructs a complete training process through speech input with noise and mask, as well as extended character sequences. This method not only simplifies the design of the model, but also provides effective training signals for subsequent speech generation tasks. Through this process, the model can better learn the mapping relationship between text and speech and achieve high-quality speech synthesis.

[0088] In the embodiment of the present application, in order to generate speech with the required content, the present application uses the Mel spectrogram feature x of the audio prompt ref , its transcription text y ref and text prompty gen The audio cues are used to provide speaker characteristics, while the text cues are used to guide the content of the generated speech.

[0089] To sample from the learned distribution, the transformed Mel feature xref , and the concatenated and expanded character sequence z ref·gen As a condition. This application has:

[0090] 1. Condition input:

[0091] Condition=(x ref , z ref·gen )

[0092] Among them, x ref is the Mel-spectrogram feature of the audio cue, z ref·gen It is a concatenated and expanded sequence of characters.

[0093] 2. Sampling process:

[0094] The process of sampling and generating speech from the learned distribution can be expressed as follows:

[0095] x gen ~P(x|x ref , z ref·gen )

[0096] Among them, x gen is the mel-spectrogram of the generated speech.

[0097] 3. Post-processing:

[0098] To convert the generated Mel spectrogram into an audio signal, the Inverse Mel Transform can be used:

[0099] Audio=InverseMel(x gen )

[0100] This step converts the generated Mel-spectrogram into an audible audio signal.

[0101] In the inference phase, the application generates speech with the required content and speaker characteristics by using a combination of audio and text prompts. Through simple duration estimation and conditional sampling, the application is able to achieve high-quality speech generation while maintaining the simplicity of the model. The generated Mel spectrogram is converted into an audio signal through an inverse Mel transform, and the speech generation process is finally completed.

[0102] from Figure 3 It can be seen that the application starts with a sampled noise x0, and the goal is to get the other end of the flow process x1. Therefore, the application uses an ordinary differential equation (ODE) solver to gradually integrate from ψ0(x0) = x0 to ψ1(x0) = x1, and its derivative is During inference, stream steps are provided in an ordered manner, for example, a certain number of steps are uniformly sampled from 0 to 1 according to the setting of NFE (Number of Function Evaluations).

[0103] In the embodiment of the present application, after obtaining the model v t After the Mel spectrum generated by the 0DE solver, this application discards x ref Then, the application uses a vocoder to convert the mel-spectrogram back into a speech signal.

[0104] In the embodiment of the present application, the reasoning process starts with the sampled noise and gradually generates the target Mel spectrogram x1 through the 0DE solver. During the generation process, the present application provides the flow steps in an orderly manner according to the NFE settings. After the generation is completed, the present application discards the audio prompt part and uses the vocoder to convert the Mel spectrogram into the final speech signal.

[0105] In the embodiment of the present application, the flow step t of CFM is used as the condition of adaLN-zero, rather than being attached to the concatenated input sequence as in Voicebox. The present application found that the additional mean pooling token of the text sequence is not necessary for the adaLN condition, probably because the TTS task requires more stringent guidance, and the mean pooled text tokens are too coarse.

[0106] In some optional implementations of the embodiments of the present application, the present application adopts some position embedding settings in Voicebox. The stream step is encoded using sinusoidal position embedding, and the concatenated input sequence is added with convolutional position embedding. The present application applies rotational position embedding (RoPE) to self-attention instead of symmetric bidirectional ALiBi bias. For the expanded character sequence, the present application also adds absolute sinusoidal position embedding before inputting it into the Con v NeXt module.

[0107] In an embodiment of the present application, a non-autoregressive speech synthesis method is provided, comprising: receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least audio prompt data and text prompt data; obtaining a Mel-spectrogram feature x according to the audio prompt data; ref and the transcript ref ; According to the filler and Mel spectrum feature x ref Perform character completion operation on the text prompt data to obtain the completed text y gen ; According to the completed text y gen and the transcript ref Get the sequence length N; for the transcription text y ref And complete the text y genPerform the splicing operation to obtain the spliced ​​text z ref·gen ; Mel-spectrogram feature x ref And the concatenated text z ref·gen Input the trained DiT diffusion model for diffusion operation to obtain diffusion noise data; input the diffusion noise data and sequence length N into the ordinary differential equation solver for solution operation to obtain the synthetic Mel spectrum graph feature Synthesize Mel-spectrogram features based on vocoder An inverse Mel transform operation is performed to obtain the target synthesized audio. Compared with the prior art, this application constructs a non-autoregressive model based on stream matching and diffusion transformer, which can more effectively utilize the start information through a concise process, advanced model architecture and efficient sampling strategy, thereby improving the quality and accuracy of the generated speech.

[0108] In some optional implementations of the embodiments of the present application, the above-mentioned method of completing the text y gen and the transcript ref The step of obtaining the sequence length N specifically includes the following steps:

[0109] According to the completed text y gen and the transcript ref The sequence length is estimated based on the ratio of the number of characters to obtain the sequence length N.

[0110] In some optional implementations of the embodiments of the present application, the above-mentioned method of completing the text y gen and the transcript ref The step of obtaining the sequence length N specifically includes the following steps:

[0111] Call the trained sequence length prediction model and complete the text y gen and the transcript ref Input into the sequence length prediction model to perform sequence length prediction operation to obtain the sequence length N.

[0112] In the present embodiment, the sequence length N (i.e., time length) now becomes a key factor, and the model needs to be informed of the length required to generate samples. The present application can train a separate model based on x ref ,y ref and gen To predict and provide the duration. Here, this application simply calculates the duration based on y gen and ref The duration is estimated by the ratio of the number of characters in the . This application assumes that the total length of the characters does not exceed the length of the mel-spectrogram, so fillers are used as in training.

[0113] In some optional implementations of the embodiments of the present application, when the transcribed text yref And complete the text y gen Perform the splicing operation to obtain the spliced ​​text z ref·gen Before the step of:

[0114] Will fill in the text y gen Input into ConvNeXtV2 convolutional neural network for feature enhancement operation to obtain enhanced text;

[0115] Transcript y ref And complete the text y gen Perform the splicing operation to obtain the spliced ​​text z ref·gen The steps specifically include the following steps:

[0116] Transcript y ref And the enhanced text is spliced ​​to obtain the spliced ​​text z ref·gen .

[0117] In the embodiment of the present application, in order to enhance the text alignment capability of the model, the present application also introduces the Co nvNeXtV2 module.

[0118] In an embodiment of the present application, the input of the model includes a character sequence, noisy speech, and masked speech. Before the feature dimension is spliced, the character sequence is first processed separately by a ConvNeXt module. Experiments show that this method provides an independent modeling space for text input, enabling it to better prepare itself in subsequent contextual learning. Unlike the mandatory phoneme-level alignment in Voicebox, the present application does not explicitly introduce strict boundaries for the text, but allows semantic and acoustic features to be learned jointly throughout the model. In addition, unlike the input with large length differences (different effective information lengths) used in E2TTS, the design of the present application alleviates this length difference.

[0119] In some optional implementations of the embodiments of the present application, when the Mel frequency spectrum feature x ref , splice text z ref·gen And the sequence length N is input into the trained DiT diffusion model for diffusion operation to obtain the diffusion noise data, and the following steps are also included:

[0120] Performing a swing sampling operation on the diffusion noise data according to the swing function S(t) to obtain the swing sampling noise data;

[0121] Input the diffuse noise data and sequence length N into the ordinary differential equation solver for solving the problem and obtain the synthetic Mel spectrum feature. The steps specifically include the following steps:

[0122] Input the oscillating sampling noise data and sequence length N into the ordinary differential equation solver for solving the operation to obtain the synthetic Mel spectrum feature

[0123] In the embodiments of the present application, the present application proposes a "swing sampling" strategy during reasoning, which significantly improves the performance and efficiency of the model. This sampling strategy can be easily applied to the existing stream matching-based model without retraining. The design of the present application not only speeds up the training speed, but also greatly improves the reasoning speed compared to the existing diffusion-based TTS model.

[0124] In the embodiment of the present application, CFM can be regarded as a model for v prediction using cosine scheduling. In image synthesis, a unimodal logit-normal sampling strategy is proposed to further schedule the flow steps so that the intermediate flow steps are sampled more frequently, thereby giving them greater weight. The present application speculates that this sampling method can make the learning difficulty of the model more evenly distributed across different flow steps t∈[0,1].

[0125] In contrast, the model of this application uses traditional uniform sampling t~U[0,1] during training, but applies a non-uniform sampling during inference. Specifically, this application defines a "swing sampling" function:

[0126] Swing sampling function

[0127] 1. Definition:

[0128] The swing sampling function S(t) is used to generate stream step t at inference time. This function dynamically adjusts the sampling density according to the generation stage of the sample.

[0129] This application defines S(t) as follows:

[0130]

[0131] Where μ is the peak position, σ controls the width of the peak, and Z is a normalization constant that ensures that S(t) is a valid probability distribution.

[0132] 2. Sampling process:

[0133] In the inference process, this application uses S(t) to sample the flow step t. The specific steps are as follows:

[0134] First, a flow step t is sampled according to S(t).

[0135] This t is then used to perform the inference step of the model.

[0136] This approach allows the application to perform non-uniform sampling of different flow steps at inference time, thereby more effectively utilizing the generative power of the model.

[0137] 3. Parameter adjustment:

[0138] The values ​​of μ and σ can be adjusted according to the specific task requirements. For example, in the stage where more details need to be generated, μ can be set to 0.5 and σ can be reduced to concentrate the sampling in the middle steps.

[0139] In the embodiment of the present application, by adopting the swing sampling strategy, the model of the present application can more flexibly control the sampling distribution of the flow steps during reasoning, thereby improving efficiency while maintaining the generation quality. This method is not only applicable to the model of the present application, but can also be easily applied to other CFM-based models to further improve their generation performance and efficiency.

[0140] In the embodiment of the present application, the CFM model is more concerned with outlining the outline of speech from pure noise in the early stage (when t→0), and more focused on modifying the details in the later stage. Therefore, the alignment of speech and text will be determined based on the first few generation results. By introducing a scaling parameter S<0, the present application enables the model to perform more reasoning when the t value is small, thereby providing more starting information for the ODE solver to perform more accurate evaluation in the initial integration step.

[0141] In the embodiment of the present application, this strategy enables the model to better capture the overall outline of speech in the early stages of speech generation, while focusing on the perfection of details in the later stages. By adjusting the value of t, the model can more effectively utilize the starting information, thereby improving the quality and accuracy of the generated speech.

[0142] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0143] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0145] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0146] Further references Figure 4 , as a response to the above Figure 2 The present application provides an embodiment of a non-autoregressive speech synthesis device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0147] like Figure 4 As shown, the non-autoregressive speech synthesis device 200 of the embodiment of the present application includes:

[0148] The request receiving module 210 is used to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least audio prompt data and text prompt data;

[0149] The audio processing module 220 is used to obtain the Mel-spectrogram feature x according to the audio prompt data. ref and the transcript ref ;

[0150] The character completion module 230 is used to complete the character according to the filler and the Mel spectrum feature x ref Perform character completion operation on the text prompt data to obtain the completed text y gen ;

[0151] The sequence length acquisition module 240 is used to complete the text y genand the transcript ref Get the sequence length N;

[0152] The splicing module 250 is used to transcribe the text y ref And complete the text y gen Perform the splicing operation to obtain the spliced ​​text z ref·gen ;

[0153] Diffusion module 260, used to transform the Mel frequency spectrum feature x ref And the concatenated text z ref·gen Input into the trained DiT diffusion model for diffusion operation to obtain diffusion noise data;

[0154] The ordinary differential equation solving module 270 is used to input the diffuse noise data and the sequence length N into the ordinary differential equation solver for solving the problem, and obtain the synthetic mel spectrum feature.

[0155] Inverse Mel transform module 280, for synthesizing Mel spectrogram features according to the vocoder Perform an inverse Mel transform operation to obtain the target synthesized audio.

[0156] In an embodiment of the present application, a non-autoregressive speech synthesis device 200 is provided, comprising: a request receiving module 210, configured to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least audio prompt data and text prompt data; an audio processing module 220, configured to obtain a Mel-spectrogram feature x according to the audio prompt data ref and the transcript ref ; Character completion module 230, for filling in the Mel spectrum feature x ref Perform character completion operation on the text prompt data to obtain the completed text y gen ; Sequence length acquisition module 240, used to fill in the text y gen and the transcript ref Get the sequence length N; splicing module 250 is used to transcribe the text y ref And complete the text y gen Perform the splicing operation to obtain the spliced ​​text z ref·gen ; Diffusion module 260, used to transform the Mel frequency spectrum feature x ref And the concatenated text z ref·gen Input the trained DiT diffusion model to perform diffusion operation to obtain diffusion noise data; the ordinary differential equation solving module 270 is used to input the diffusion noise data and the sequence length N into the ordinary differential equation solver for solving operation to obtain the synthetic Mel spectrum feature Inverse Mel transform module 280, for synthesizing Mel spectrogram features according to the vocoder An inverse Mel transform operation is performed to obtain the target synthesized audio. Compared with the prior art, this application constructs a non-autoregressive model based on stream matching and diffusion transformer, which can more effectively utilize the start information through a concise process, advanced model architecture and efficient sampling strategy, thereby improving the quality and accuracy of the generated speech.

[0157] In some optional implementations of the embodiments of the present application, the sequence length acquisition module includes:

[0158] The length estimation submodule is used to complete the text y gen and the transcript ref The sequence length is estimated based on the ratio of the number of characters to obtain the sequence length N.

[0159] To solve the above technical problems, the present application also provides a computer device. Figure 5 , Figure 5 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0160] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0161] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0162] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (SmartMediaCard, SMC), a secure digital (SecureDigital, SD) card, a flash card (FlashCard), etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions of the non-autoregressive speech synthesis method, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0163] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions stored in the memory 310 or process data, such as computer-readable instructions for running the non-autoregressive speech synthesis method.

[0164] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0165] The computer device provided in this application constructs a non-autoregressive model based on stream matching and diffusion transformer. Through a concise process, advanced model architecture and efficient sampling strategy, it can more effectively utilize the starting information, thereby improving the quality and accuracy of the generated speech.

[0166] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the non-autoregressive speech synthesis method as described above.

[0167] The computer-readable storage medium provided in the present application constructs a non-autoregressive model based on flow matching and diffusion transformer. Through a concise process, advanced model architecture and efficient sampling strategy, it can more effectively utilize the starting information, thereby improving the quality and accuracy of the generated speech.

[0168] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0169] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A non-autoregressive speech synthesis method, characterized in that: The steps include: Receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least audio prompt data and text prompt data; Obtain the Mel-spectrogram feature x according to the audio prompt data ref and the transcript ref ; According to the filler and the Mel-spectrogram feature x ref Perform character completion operation on the text prompt data to obtain the completed text y ger ; Complete the text y according to gen and the transcribed text y ref Get the sequence length N; The transcribed text y ref And the completed text y gen Perform the splicing operation to obtain the spliced ​​text z ref·gen ; The Mel-spectrogram feature x ref , the concatenated text z ref·gen The sequence length N is input into the trained DiT diffusion model to perform diffusion operation to obtain diffusion noise data; The diffuse noise data is input into the ordinary differential equation solver for solving, and the synthesized Mel spectrum feature is obtained. According to the vocoder, the synthesized Mel-spectrogram features Perform an inverse Mel transform operation to obtain the target synthesized audio.

2. The non-autoregressive speech synthesis method according to claim 1, characterized in that: The method according to the completion text y gen and the transcribed text y ref The step of obtaining the sequence length N specifically includes the following steps: Complete the text y according to gen and the transcribed text y ref The sequence length is estimated based on the ratio of the number of characters to obtain the sequence length N.

3. The non-autoregressive speech synthesis method according to claim 1, characterized in that: The method according to the completion text y gen and the transcribed text y ref The step of obtaining the sequence length N specifically includes the following steps: Call the trained sequence length prediction model and fill in the text y gen and the transcribed text y ref The sequence length is input into the sequence length prediction model to perform a sequence length prediction operation to obtain the sequence length N.

4. The non-autoregressive speech synthesis method according to claim 1, characterized in that: In the transcription of the ref And the completed text y gen Perform the splicing operation to obtain the spliced ​​text z ref·gen Before the step of: The padded text y gen Input into ConvNeXtV2 convolutional neural network for feature enhancement operation to obtain enhanced text; The transcribed text y ref And the completed text y gen Perform the splicing operation to obtain the spliced ​​text z ref·gen The steps specifically include the following steps: The transcribed text y ref And the enhanced text is spliced ​​to obtain the spliced ​​text z ref·gen .

5. The non-autoregressive speech synthesis method according to claim 1, characterized in that: In the Mel-spectrogram feature x ref , the concatenated text z ref·gen After the step of inputting the sequence length N into the trained DiT diffusion model for diffusion operation to obtain diffusion noise data, the following steps are also included: Performing a swing sampling operation on the diffusion noise data according to the swing function S(t) to obtain swing sampling noise data; The diffuse noise data and the sequence length N are input into the ordinary differential equation solver for solving the problem, and the synthesized Mel spectrum feature is obtained. The steps specifically include the following steps: The swing sampling noise data and the sequence length N are input into the ordinary differential equation solver for solving, and the synthetic Mel spectrum feature is obtained.

6. The non-autoregressive speech synthesis method according to claim 5, characterized in that: The swing function S(t) is expressed as: Among them, μ represents the peak position, σ represents the width of the controlled peak, and S represents the normalization constant to ensure that S(t) is a valid probability distribution.

7. A non-autoregressive speech synthesis device, characterized in that: include: A request receiving module, used to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes at least audio prompt data and text prompt data; An audio processing module, used to obtain a Mel-spectrogram feature x according to the audio prompt data ref and the transcript ref ; The character completion module is used to fill in the character according to the filler and the Mel spectrum feature x ref Perform character completion operation on the text prompt data to obtain the completed text y gen ; A sequence length acquisition module is used to complete the text y according to the gen and the transcribed text y ref Get the sequence length N; A splicing module is used to transcribe the text y ref And the completed text y gen Perform the splicing operation to obtain the spliced ​​text z ref·gen ; Diffusion module, used to transform the Mel frequency spectrum feature x ref And the concatenated text z ref·gen Input into the trained DiT diffusion model for diffusion operation to obtain diffusion noise data; An ordinary differential equation solving module is used to input the diffuse noise data and the sequence length N into the ordinary differential equation solver for solving the problem, and obtain the synthetic Mel spectrum feature. An inverse Mel transform module is used to transform the synthesized Mel spectrogram features according to the vocoder Perform an inverse Mel transform operation to obtain the target synthesized audio.

8. The non-autoregressive speech synthesis device according to claim 7, characterized in that: The sequence length acquisition module comprises: The length estimation submodule is used to complete the text y according to the gen and the transcribed text y ref The sequence length is estimated based on the ratio of the number of characters to obtain the sequence length N.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the non-autoregressive speech synthesis method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the non-autoregressive speech synthesis method according to any one of claims 1 to 6.

Images