Speech synthesis method based on autoregression potential diffusion and related equipment
By adopting the potential diffusion method of autoregression in speech synthesis technology, discrete semantic prediction and acoustic feature extraction, the problem of quantization noise in traditional technology is solved, and higher speech synthesis quality and efficiency are achieved.
Patent Information
- Application Number
- CN202411793973.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-05
- Publication Date
- 2025-03-11
AI Technical Summary
Traditional speech synthesis techniques introduce quantization noise when performing discrete modeling on continuous domains, reducing reconstruction quality and fidelity.
The speech synthesis method based on potential diffusion of autoregression is adopted, discrete semantic prediction is performed through the T2S model, acoustic feature extraction is performed using autoregressive semantics to acoustic models and non-autoregressive semantics to acoustic models, and speech synthesis is performed through VAE decoder.
It effectively improves the performance and efficiency of the speech synthesis model, reduces quantization noise, and improves reconstruction quality and fidelity.
Smart Images
Figure CN119600984A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of speech synthesis, and in particular to a speech synthesis method based on autoregressive potential diffusion and related equipment. Background Art
[0002] Autoregressive (AR) models are often associated with discrete representations, probably because of the successful application of large language models (LLMs) on discrete modalities. Inspired by the success of LLMs, continuous modalities such as audio and images are often quantized into discrete forms for processing. In the field of audio generation, researchers often use residual vector quantization (RVQ), a process that gradually refines the approximation by quantizing the residual. The resulting discrete codes are used for discrete autoregressive modeling.
[0003] However, the applicant found that discrete modeling on the continuous domain requires quantization, which degrades the reconstruction quality and limits the upper limit of fidelity. Although using multiple RVQ quantizers can improve fidelity, fine RVQ codes may cause quantization noise, which is detrimental to discrete modeling methods. Summary of the invention
[0004] The purpose of the embodiments of the present application is to propose a speech synthesis method based on autoregressive potential diffusion and related equipment to solve the problem that traditional technical means will cause quantization noise.
[0005] In order to solve the above technical problems, the embodiment of the present application provides a speech synthesis method based on autoregressive potential diffusion, which adopts the following technical solution:
[0006] Receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes input text data and speaker prompt data;
[0007] Performing a discrete semantic prediction operation on the input text data and the speaker prompt data according to the T2S model to obtain discrete semantic features;
[0008] performing an acoustic feature extraction operation on the discrete semantic features and the speaker prompt data according to a semantic-to-acoustic model to obtain acoustic feature data;
[0009] A speech synthesis operation is performed on the acoustic feature data according to the VAE decoder to obtain target speech data.
[0010] Furthermore, the semantic-to-acoustic model is an autoregressive semantic-to-acoustic model, wherein:
[0011] The training objective of the autoregressive semantic-to-acoustic model is expressed as:
[0012]
[0013] Among them, x i represents the i-th acoustic feature, ω represents the discrete semantic feature, and s represents the speaker prompt data.
[0014] Furthermore, the loss function of the autoregressive semantics to acoustic model is expressed as:
[0015]
[0016] Among them, z i-1 represents the i-1th context condition vector.
[0017] Furthermore, the context condition vector is expressed as:
[0018] z i ,...,z n =Θ(|ω,s,x 1 ,...,x n )
[0019] Among them, Θ represents the Transformer model.
[0020] Furthermore, the semantic-to-acoustic model is a non-autoregressive semantic-to-acoustic model, and the step of performing an acoustic feature extraction operation on the discrete semantic features and the speaker prompt data according to the semantic-to-acoustic model to obtain acoustic feature data specifically includes the following steps:
[0021] Prompt the speaker with data x i Perform a random mask operation to obtain the masked acoustic feature r i ;
[0022] According to the acoustic feature r after the mask i Generate context vector z n ;
[0023] According to the context vector z n and the diffuse head predicts the next acoustic signature x i+1 ;
[0024] When the speaker prompts data x i After all predictions are made, the acoustic feature data is obtained. Further, the masked acoustic feature r i It is expressed as:
[0025] r i =m i q+(1-m i )·x i
[0026] Among them, r irepresents the acoustic features after masking, m i represents the mask indicator, q represents the fixed, learnable embedding, and x i Represents the original acoustic features.
[0027] Furthermore, the loss function of the non-autoregressive semantics to acoustic model is expressed as:
[0028]
[0029] in, represents the predicted acoustic features, x i Represents the original acoustic features.
[0030] In order to solve the above technical problems, the embodiment of the present application further provides a speech synthesis device based on autoregressive potential diffusion, which adopts the following technical solution:
[0031] A request receiving module, configured to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes input text data and speaker prompt data;
[0032] A discrete semantic prediction module, used for performing a discrete semantic prediction operation on the input text data and the speaker prompt data according to a T2S model to obtain discrete semantic features;
[0033] An acoustic feature extraction module, used for performing an acoustic feature extraction operation on the discrete semantic features and the speaker prompt data according to a semantic-to-acoustic model to obtain acoustic feature data;
[0034] The speech synthesis module is used to perform a speech synthesis operation on the acoustic feature data according to the VAE decoder to obtain target speech data.
[0035] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0036] The invention comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned speech synthesis method based on autoregressive potential diffusion when executing the computer-readable instructions.
[0037] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0038] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech synthesis method based on autoregressive potential diffusion as described above are implemented.
[0039] The present application provides a speech synthesis method based on autoregressive potential diffusion, comprising: receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes input text data and speaker prompt data; performing discrete semantic prediction operations on the input text data and the speaker prompt data according to a T2S model to obtain discrete semantic features; performing acoustic feature extraction operations on the discrete semantic features and the speaker prompt data according to a semantic-to-acoustic model to obtain acoustic feature data; performing speech synthesis operations on the acoustic feature data according to a VAE decoder to obtain target speech data. Compared with the prior art, the present application effectively improves the performance and efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0041] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0042] Figure 2 is a flow chart for implementing a speech synthesis method based on autoregressive potential diffusion provided in an embodiment of the present application;
[0043] Figure 3 is a structural schematic diagram of a speech synthesis device based on autoregressive latent diffusion provided in an embodiment of the present application;
[0044] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0046] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0047] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0048] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.
[0049] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0050] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.
[0051] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .
[0052] It should be noted that the speech synthesis method based on autoregressive latent diffusion provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the speech synthesis device based on autoregressive latent diffusion is generally arranged in the server / terminal device.
[0053] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0054] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech synthesis method based on autoregressive latent diffusion according to the present application. The speech synthesis method based on autoregressive latent diffusion comprises: step S201, step S202, step S203 and step S204.
[0055] In step S201, a speech synthesis request sent by a user terminal is received, wherein the speech synthesis request includes input text data and speaker prompt data;
[0056] In step S202, a discrete semantic prediction operation is performed on the input text data and the speaker prompt data according to the T2S model to obtain discrete semantic features;
[0057] In step S203, an acoustic feature extraction operation is performed on the discrete semantic features and the speaker prompt data according to the semantic-to-acoustic model to obtain acoustic feature data;
[0058] In step S204, a speech synthesis operation is performed on the acoustic feature data according to the VAE decoder to obtain target speech data.
[0059] In an embodiment of the present application, speech synthesis based on autoregressive potential diffusion is proposed, which is a feature-by-feature potential diffusion model for continuous representation of zero-sample text to speech. The present application realizes the generation of variable-length outputs, which solves the problem that does not exist in the image generation method because the number of features generated in image generation is fixed. The present application uses semantic features - quantized embeddings of self-supervised models - to provide contextual information and define generation stop conditions. It does not rely on text-audio alignment, which makes it easier to utilize large data sources and can synthesize the audio of the target speaker with a three-second speaker prompt.
[0060] In the embodiment of the present application, the present solution realizes zero-shot text-to-speech synthesis, which can generate speech based on given text and speaker prompts. It achieves this by predicting the continuous VAE latent variable x through a feature-by-feature diffusion head. The method of the present application uses the semantic feature ω as an auxiliary representation to provide contextual information and determine the stopping condition.
[0061] In the embodiment of the present application, the speech synthesis task is divided into two subtasks: text to semantics (T2S) and semantics to acoustics (S2A), and each subtask is processed by a different model. The T2S model uses a causal Transformer to autoregressively predict discrete semantic features based on text and speaker information:
[0062] ω i =T2S(x 1:i-1 ,S)
[0063] Among them, ω i represents the i-th semantic feature, x 1:i-1 represents the first i-1 text features, and S represents the speaker information.
[0064] In the embodiment of the present application, in the S2A stage, the present application uses a feature-by-feature diffusion model to generate acoustic features based on the predicted semantic features. Specifically, the model predicts the acoustic feature y corresponding to each semantic feature. i To implement speech synthesis:
[0065] y i =S2A(ω i )
[0066] Among them, y i is related to the semantic feature ω i The corresponding acoustic characteristics.
[0067] In the embodiment of the present application, the T2S model determines the length of the synthesized audio. Then, the S2A model of the present application predicts acoustic features based on semantic features, using an additional Transformer model and diffusion head to model the continuous distribution. The present application provides two variants of semantic-to-acoustic models: autoregressive (AR) and non-autoregressive (NAR), which follows the relevant literature on discrete acoustic feature modeling.
[0068] In an embodiment of the present application, a speech synthesis method based on autoregressive potential diffusion is provided, comprising: receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes input text data and speaker prompt data; performing discrete semantic prediction operations on the input text data and speaker prompt data according to a T2S model to obtain discrete semantic features; performing acoustic feature extraction operations on the discrete semantic features and speaker prompt data according to a semantic-to-acoustic model to obtain acoustic feature data; performing speech synthesis operations on the acoustic feature data according to a VAE decoder to obtain target speech data. Compared with the prior art, the present application effectively improves the performance and efficiency of the model.
[0069] In some optional implementations of the embodiments of the present application, the above semantic-to-acoustic model is an autoregressive semantic-to-acoustic model, wherein:
[0070] The training objective of the autoregressive semantic-to-acoustic model is expressed as:
[0071]
[0072] Among them, x i represents the i-th acoustic feature, ω represents the discrete semantic feature, and s represents the speaker prompt data.
[0073] In some optional implementations of the embodiments of the present application, during the training process, the present application inputs the potential frame, semantic features and speaker prompts into a causal Transformer to obtain a contextual condition vector:
[0074] z i ,...,z n =Θ(|ω,s,x 1 ,...,x n )
[0075] Among them, Θ represents the Transformer model.
[0076] Each acoustic frame x i The prediction is based on the previous context condition vector z i-1 , the loss function of this application is:
[0077]
[0078] Among them, z i-1 represents the i-1th context condition vector.
[0079] In an embodiment of the present application, this loss function encourages the model to correctly predict the probability distribution of the current acoustic frame given the previous context conditions.
[0080] In some optional implementations of the embodiments of the present application, the semantic-to-acoustic model is a non-autoregressive semantic-to-acoustic model, and the step of performing an acoustic feature extraction operation on discrete semantic features and speaker prompt data according to the semantic-to-acoustic model to obtain acoustic feature data specifically includes the following steps:
[0081] Prompt the speaker with data x i Perform a random mask operation to obtain the masked acoustic feature r i ;
[0082] According to the acoustic features r after masking i Generate context vector z n ;
[0083] According to the context vector z n and the diffuse head predicts the next acoustic signature x i+1;
[0084] When the speaker prompts data x i After prediction, the acoustic feature data is obtained.
[0085] In an embodiment of the present application, MaskGIT trains a bidirectional Transformer using a discrete masked language modeling objective. During inference, it gradually unmasks the mask through K inference steps, each step based on previously predicted features. SoundStorm extends this process to predict RVQ codes based on semantic features by applying the MaskGIT process to each RVQ layer. The present application further extends the MaskGIT method, using a diffusion head to predict continuous acoustic features based on semantic features.
[0086] In the embodiment of the present application, given a scheduling function γ(r):[0,1]→[0,1] and a sequence x 1 ,...,x n , this application samples r from the uniform distribution U[0,1] and masks out γ(r)·n acoustic features. Define the random mask indicator m=(m 1 ,...,m n ), where m i ∈{0, 1}. This application replaces the acoustic features of the mask with a fixed, learnable embedding q and defines:
[0087] r i =m i q+(1-m i )·x i
[0088] Similar to SoundStorm, a semantic feature ω i and the masked acoustic feature r i is embedded into a single Transformer feature, while the speaker hint feature remains unmasked and appended as context. The final sequence is input into a bidirectional Transformer, and the resulting context vector z i ,...,z n Used to predict masked acoustic features using a diffusion head. The loss function is the denoising of the masked acoustic features.
[0089] 1. Random masking process:
[0090] r i =m i q+(1-m i )·x i
[0091] Among them, r irepresents the acoustic features after masking, m i represents the mask indicator, q represents the fixed, learnable embedding, and x i Represents the original acoustic features.
[0092] 2. Context vector generation:
[0093] -Semantic featuresω i and the masked acoustic feature r i is embedded and input into the bidirectional Transformer to generate a context vector:
[0094]
[0095] in, Represents a feature concatenation operation.
[0096] 3. Diffusion head prediction:
[0097] Using context vector and the diffuse head predicts the next acoustic signature x i+1 :
[0098]
[0099] 4. Loss function:
[0100] The loss function is the denoising loss of the masked acoustic features:
[0101]
[0102] in, represents the predicted acoustic features, x i Represents the original acoustic features.
[0103] In an embodiment of the present application, during the inference phase, MaskGIT usually selects the feature with the highest confidence for prediction. However, in the continuous diffusion model, it is not easy to accurately measure confidence. Therefore, the present application chooses to randomly select the features to be unmasked. As for the confidence-based selection method in MaskGIT reasoning, it is concluded that random unmasking is unexpectedly better than confidence-based unmasking, which verifies the design choice of the present application. Specifically, given a sequence containing n features, in K MaskGIT steps, each step k∈[K], the present application randomly selects n·γ(k)-γ(k-1) additional features for unmasking, and uses a feature-by-feature diffusion head for prediction.
[0104] In the embodiments of this application, all models use a common discrete architecture, only replacing the input projection and prediction head, so we can implement a discrete variant. The discrete method uses the RVQ-GAN quantizer to generate a series of discrete codes (q1 ,...,q Q ). These codes are predicted from the context embedding z via MLP prediction heads, one for each codebook. The S2A-AR discrete model predicts all codes in parallel using the delay pattern proposed by MusicGen. The S2A-NAR discrete model implements SoundStorm, which applies Q MaskGIT processes to each codebook. Unlike the original SoundStorm implementation, we replace confidence-based unmasking with random unmasking as we find that random unmasking works better. Given K MaskGIT steps, SoundStorm requires QK passes through the Transformer because it applies a MaskGIT process to each RVQ layer, while this proposal only requires K Transformer passes. The T2A discrete model predicts semantic and acoustic tokens directly from the text in parallel, using the delay pattern described above. We compare it with a commercial SOTA zero-shot text-to-speech model in a listening test. It is found that the performance and efficiency of the model are further improved.
[0105] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0106] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0107] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0108] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0109] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a speech synthesis device based on autoregressive potential diffusion, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0110] like Figure 3 As shown, the speech synthesis device 200 based on autoregressive potential diffusion of the embodiment of the present application includes:
[0111] A request receiving module 210, configured to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes input text data and speaker prompt data;
[0112] A discrete semantic prediction module 220 is used to perform discrete semantic prediction operations on input text data and speaker prompt data according to the T2S model to obtain discrete semantic features;
[0113] An acoustic feature extraction module 230, configured to perform an acoustic feature extraction operation on discrete semantic features and speaker prompt data according to a semantic-to-acoustic model to obtain acoustic feature data;
[0114] The speech synthesis module 240 is used to perform a speech synthesis operation on the acoustic feature data according to the VAE decoder to obtain target speech data.
[0115] In an embodiment of the present application, a speech synthesis device 200 based on autoregressive potential diffusion is provided, including: a request receiving module 210, for receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes input text data and speaker prompt data; a discrete semantic prediction module 220, for performing discrete semantic prediction operations on the input text data and speaker prompt data according to a T2S model to obtain discrete semantic features; an acoustic feature extraction module 230, for performing acoustic feature extraction operations on discrete semantic features and speaker prompt data according to a semantic-to-acoustic model to obtain acoustic feature data; and a speech synthesis module 240, for performing speech synthesis operations on acoustic feature data according to a VAE decoder to obtain target speech data. Compared with the prior art, the present application effectively improves the performance and efficiency of the model.
[0116] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.
[0117] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.
[0118] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0119] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions of the speech synthesis method based on autoregressive potential diffusion, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.
[0120] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions or process data stored in the memory 310, such as computer-readable instructions for running the speech synthesis method based on autoregressive potential diffusion.
[0121] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.
[0122] The computer equipment provided by this application effectively improves the performance and efficiency of the model.
[0123] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the speech synthesis method based on autoregressive potential diffusion as described above.
[0124] The computer-readable storage medium provided in this application effectively improves the performance and efficiency of the model.
[0125] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0126] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A speech synthesis method based on autoregressive latent diffusion, characterized in that: The steps include: Receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes input text data and speaker prompt data; Performing a discrete semantic prediction operation on the input text data and the speaker prompt data according to the T2S model to obtain discrete semantic features; performing an acoustic feature extraction operation on the discrete semantic features and the speaker prompt data according to a semantic-to-acoustic model to obtain acoustic feature data; A speech synthesis operation is performed on the acoustic feature data according to the VAE decoder to obtain target speech data.
2. The method for speech synthesis based on autoregressive latent diffusion according to claim 1, characterized in that: The semantic-to-acoustic model is an autoregressive semantic-to-acoustic model, where: The training objective of the autoregressive semantic-to-acoustic model is expressed as: Among them, x i represents the i-th acoustic feature, ω represents the discrete semantic feature, and s represents the speaker prompt data.
3. The method for speech synthesis based on autoregressive latent diffusion according to claim 1, characterized in that: The loss function of the autoregressive semantic to acoustic model is expressed as: Among them, z i-1 represents the i-1th context condition vector.
4. The method for speech synthesis based on autoregressive latent diffusion according to claim 3, characterized in that: The context condition vector is expressed as: z i ,...,z n =Θ(|ω,s,x1,...,x n ) Among them, Θ represents the Transformer model.
5. The method for speech synthesis based on autoregressive latent diffusion according to claim 1, characterized in that: The semantic-to-acoustic model is a non-autoregressive semantic-to-acoustic model, and the step of performing an acoustic feature extraction operation on the discrete semantic features and the speaker prompt data according to the semantic-to-acoustic model to obtain acoustic feature data specifically includes the following steps: Prompt data x to the speaker i Perform a random mask operation to obtain the masked acoustic feature r i ; According to the acoustic feature r after the mask i Generate context vector z n ; According to the context vector z n and the diffuse head predicts the next acoustic signature x i+1 ; When the speaker prompts data x i After prediction is performed, the acoustic feature data is obtained.
6. The method for speech synthesis based on autoregressive latent diffusion according to claim 5, characterized in that: The masked acoustic feature r i It is expressed as: r i =m i ·q+(1-m i )·x i Among them, r i represents the acoustic features after masking, m i represents the mask indicator, q represents the fixed, learnable embedding, and x i Represents the original acoustic features.
7. The method for speech synthesis based on autoregressive latent diffusion according to claim 5, characterized in that: The loss function of the non-autoregressive semantic to acoustic model is expressed as: in, represents the predicted acoustic features, x i Represents the original acoustic features.
8. A speech synthesis device based on autoregressive latent diffusion, characterized in that: include: A request receiving module, configured to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes input text data and speaker prompt data; A discrete semantic prediction module, used for performing a discrete semantic prediction operation on the input text data and the speaker prompt data according to a T2S model to obtain discrete semantic features; An acoustic feature extraction module, used for performing an acoustic feature extraction operation on the discrete semantic features and the speaker prompt data according to a semantic-to-acoustic model to obtain acoustic feature data; The speech synthesis module is used to perform a speech synthesis operation on the acoustic feature data according to the VAE decoder to obtain target speech data.
9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the speech synthesis method based on autoregressive latent diffusion as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech synthesis method based on autoregressive latent diffusion as described in any one of claims 1 to 7.