Speech synthesis method based on codec and related equipment

Abstract

The embodiment of the invention belongs to the technical field of speech synthesis, and relates to a speech synthesis method and related equipment based on a codec, and the method comprises the steps: receiving a speech synthesis request sent by a user terminal, the speech synthesis request comprising text prompt data and reference audio data; inputting the text prompt data into a phoneme conversion module for phoneme conversion operation to obtain a phoneme sequence p; performing feature extraction operation on the reference audio data to obtain an initial acoustic condition # imgabs0 #, and inputting the phoneme sequence p and the initial acoustic condition # imgabs1 # into a speech synthesis model based on a coder and decoder to perform coding prediction operation to obtain a coding prediction sequence; and carrying out decoding operation on the coding prediction sequence to obtain a target synthetic audio. According to the method, the plurality of prediction heads predict the plurality of marks in each reasoning step of the autoregression model, so that the effect that the synthesis time is linearly reduced along with the increase of the number of the marks is realized.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to a codec-based speech synthesis method and related equipment. Background Art

[0002] LM-based TTS models take advantage of the scalability of the Transformer model, advances in audio encoders, and large speech datasets to demonstrate strong generation capabilities. This type of model can be seen as an extension of a large language model, treating speech as a new language and converting a sequence of tokens into a waveform through a decoder or vocoder module. Since speech itself is also in sequence form, leveraging the advantages of pre-trained large language models in the text field is a promising direction.

[0003] However, the applicant found that while autoregressive (AR) reasoning improves the quality of codec-based speech synthesis, it also significantly reduces the generation speed. In contrast, non-AR models can generate sentences in one step, while models using AR reasoning need to generate them step by step, with the number of steps equal to the sequence length. As the sequence length increases, the generation time also increases, and the computational complexity of the Transformer model increases quadratically with the sequence length.

[0004] It can be seen that the traditional speech synthesis method has the problem that the speed and quality cannot be effectively guaranteed. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a codec-based speech synthesis method and related equipment to solve the problem that the speed and quality of traditional speech synthesis methods cannot be effectively guaranteed.

[0006] In order to solve the above technical problems, the embodiment of the present application provides a codec-based speech synthesis method, which adopts the following technical solution:

[0007] Receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes text prompt data and reference audio data;

[0008] Inputting the text prompt data into a phoneme conversion module for phoneme conversion operation to obtain a phoneme sequence p;

[0009] Perform feature extraction on the reference audio data to obtain initial acoustic conditions

[0010] The phoneme sequence p and the initial acoustic condition Input into a codec-based speech synthesis model to perform a coding prediction operation to obtain a coding prediction sequence;

[0011] The coding prediction sequence is decoded to obtain the target synthesized audio.

[0012] Furthermore, the codec-based speech synthesis model is composed of an autoregressive module and a non-autoregressive module, wherein the phoneme sequence p and the initial acoustic condition The step of inputting into a codec-based speech synthesis model for coding prediction operation to obtain a coding prediction sequence specifically includes the following steps:

[0013] The phoneme sequence p and the initial acoustic condition Input into the autoregressive module to perform latent variable encoding operation to obtain the latent variable z 1:t , where the latent variable z 1:t It is expressed as:

[0014]

[0015] Among them, z 1:t represents the latent variable, representing the encoded information, a 1:t represents the previously generated symbol sequence, represents the initial acoustic condition from p represents the phoneme sequence, θ AR represents the parameters of the autoregressive module;

[0016] According to the latent variable z 1:t And the non-autoregressive module performs a multi-head prediction operation to obtain the coding prediction sequence.

[0017] Furthermore, according to the latent variable z 1:t The non-autoregressive module performs a multi-head prediction operation to obtain the coding prediction sequence, which specifically includes the following steps:

[0018] The multi-head prediction operation is performed according to a negative log-likelihood objective function, wherein the negative log-likelihood objective function It is expressed as:

[0019]

[0020] in, represents the multi-head prediction negative log-likelihood loss function of the autoregressive module; It means that given the phoneme sequence p and the acoustic condition and the autoregressive module parameters θ AR In the case of 1:t+n probability.

[0021] Furthermore, according to the latent variable z 1:tThe non-autoregressive module performs a multi-head prediction operation to obtain the coding prediction sequence, which specifically includes the following steps:

[0022] The coding prediction sequence obtained by the multi-head prediction operation is optimized according to the heuristic decoding technology based on the Viterbi algorithm to obtain the optimized coding prediction sequence P(a i →a j ), wherein the optimized coding prediction sequence P(a i →a j ) is expressed as:

[0023] P(a i →a j )=Q(a i ,a j )·S t (a j )

[0024] Among them, Q(a i ,a j ) represents the symbol a i Immediately after the symbol a j The previous probability, where is the transfer matrix learned from the dataset, S t (a j ) means at time t, symbol a j The probability of , which is obtained from the t prediction heads of the multi-head prediction.

[0025] Furthermore, the dimension of the transfer matrix can be obtained from Reduce to Where m≤k·n is the total number of unique symbols predicted by the top-k highest probability among the n prediction heads of the multi-head prediction.

[0026] Further, in the step of 1:t After the step of performing a multi-head prediction operation on the non-autoregressive module to obtain the coding prediction sequence, the following steps are also included:

[0027] According to the negative log-likelihood objective function The calculation results of the autoregressive module parameters θ AR Perform parameter update operation.

[0028] In order to solve the above technical problems, the embodiment of the present application further provides a speech synthesis device based on a codec, which adopts the following technical solution:

[0029] A request receiving module, configured to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes text prompt data and reference audio data;

[0030] A phoneme conversion module, used for inputting the text prompt data into the phoneme conversion module for performing a phoneme conversion operation to obtain a phoneme sequence p;

[0031] A feature extraction module is used to perform feature extraction operations on the reference audio data to obtain initial acoustic conditions

[0032] The coding prediction module is used to convert the phoneme sequence p and the initial acoustic condition Input into a codec-based speech synthesis model to perform a coding prediction operation to obtain a coding prediction sequence;

[0033] The decoding module is used to perform a decoding operation on the coded prediction sequence to obtain a target synthesized audio.

[0034] Furthermore, the codec-based speech synthesis model is composed of an autoregressive module and a non-autoregressive module, and the coding prediction module includes:

[0035] A latent variable encoding submodule is used to convert the phoneme sequence p and the initial acoustic condition Input into the autoregressive module to perform latent variable encoding operation to obtain the latent variable z 1:t , where the latent variable z 1:t It is expressed as:

[0036]

[0037] Among them, z 1:t represents the latent variable, representing the encoded information, a 1:t represents the previously generated symbol sequence, represents the initial acoustic condition from p represents the phoneme sequence, θ AR represents the parameters of the autoregressive module;

[0038] The multi-head prediction submodule is used to predict the 1:t And the non-autoregressive module performs a multi-head prediction operation to obtain the coding prediction sequence.

[0039] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0040] The invention comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the codec-based speech synthesis method as described above when executing the computer-readable instructions.

[0041] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0042] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the codec-based speech synthesis method described above are implemented.

[0043] The present application provides a codec-based speech synthesis method, comprising: receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes text prompt data and reference audio data; inputting the text prompt data into a phoneme conversion module for performing a phoneme conversion operation to obtain a phoneme sequence p; performing a feature extraction operation on the reference audio data to obtain an initial acoustic condition The phoneme sequence p and the initial acoustic condition The code is input into a codec-based speech synthesis model for coding prediction operation to obtain a coding prediction sequence; the coding prediction sequence is decoded to obtain the target synthesized audio. Compared with the prior art, the present application predicts multiple tags in each inference step of the autoregressive model through multiple prediction heads, thereby achieving the effect that the synthesis time decreases linearly with the increase in the number of tags. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0046] Figure 2 is a flow chart of an implementation of a codec-based speech synthesis method provided in an embodiment of the present application;

[0047] Figure 3 is a schematic diagram of the structure of a codec-based speech synthesis device provided in an embodiment of the present application;

[0048] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0050] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0051] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0052] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0053] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0054] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0055] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0056] It should be noted that the codec-based speech synthesis method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the codec-based speech synthesis device is generally arranged in the server / terminal device.

[0057] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0058] Continue to refer Figure 2 , shows a flow chart of an embodiment of a codec-based speech synthesis method according to the present application. The codec-based speech synthesis method comprises: step S201, step S202, step S203, step S204 and step S205.

[0059] In step S201, a speech synthesis request sent by a user terminal is received, wherein the speech synthesis request includes text prompt data and reference audio data.

[0060] In the embodiments of the present application, the user terminal refers to a terminal device used to execute the image processing method for preventing document abuse provided by the present application. The user terminal can be a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a navigation device, etc., as well as a fixed terminal such as a digital TV, a desktop computer, etc. It should be understood that the examples of user terminals here are only for the convenience of understanding and are not used to limit the present application.

[0061] In the embodiment of the present application, the user inputs a text prompt content to be synthesized through their terminal device (such as a mobile phone, computer, etc.), and this input is received by the system. This text prompt content is the specific content of the audio that the user wants the system to synthesize. Specifically, the text prompt content can be "transaction data or payment data or business data or purchase data" related to a financial institution (such as a bank, etc.). The text prompt content can also be medical data related to medical scenarios, such as personal health records, prescriptions, examination reports and other data. It should be understood that the examples of text prompt content here are only for the convenience of understanding and are not used to limit this application.

[0062] In step S202, the text prompt data is input into a phoneme conversion module for phoneme conversion to obtain a phoneme sequence p.

[0063] In step S203, feature extraction is performed on the reference audio data to obtain the initial acoustic condition

[0064] In step S204, the phoneme sequence p and the initial acoustic condition The code is input into a codec-based speech synthesis model for coding prediction operation to obtain a coding prediction sequence.

[0065] In an embodiment of the present application, the codec-based speech synthesis model workflow may be:

[0066] 1. Initialization: The model first receives the text prompt p and the initial acoustic condition

[0067] 2. Generate step by step:

[0068] At each step t, the model uses the current text prompt and the previously generated symbols a1, a2, ..., a t To predict the next symbol a t+1 .

[0069] The generated symbol a t+1 is added to the sequence of symbols for use in the next step.

[0070] 3. Termination: The generation process terminates when the model generates enough symbols or reaches a predetermined sequence length.

[0071] This step-by-step generation method enables the model to gradually generate high-quality speech signals based on text prompts and acoustic conditions. The autoregressive nature of the model enables it to fully utilize contextual information during the generation process, thereby improving the accuracy and naturalness of the generated results.

[0072] In this way, the NAR model can effectively use the phoneme sequence and the symbol sequence generated by the previous layer to iteratively generate symbols at each layer. This method not only simplifies the training process of the model, but also enables the model to gradually optimize the symbol sequence during the generation process, thereby improving the quality and efficiency of speech synthesis.

[0073] In step S205, a decoding operation is performed on the coded prediction sequence to obtain the target synthesized audio.

[0074] In the embodiment of the present application, the codec-based speech synthesis model architecture mainly consists of two modules: an autoregressive (AR) module and a non-autoregressive (NAR) module, which follows the VALL-E framework. Specifically, given a training data set containing paired data (s, p), where s represents a speech signal, p = {p1, p2, ..., pT} is the corresponding phoneme sequence, whose length is T. The codec model uses eight quantizers to compress the speech signal into discrete symbols A:

[0075] The codec model first converts the speech signal s into discrete symbols A, using eight quantizers;

[0076] The discrete symbol A contains the compressed representation of the speech signal s;

[0077] The phoneme sequence p provides a text representation of the speech signal, and its length is T;

[0078] The autoregressive module AR is used to process the generation process of speech signals;

[0079] The non-autoregressive module NAR is used to process the parallel generation of speech signals.

[0080] The overall goal of this architecture is to leverage the strengths of both autoregressive and non-autoregressive modules for efficient speech synthesis.

[0081] Specifically, for the discrete symbol sequence a1, this application trains an autoregressive Transformer decoder model θ AR The goal of the model is to predict the next symbol given the following:

[0082] 1. Text prompt p (i.e. phoneme sequence)

[0083] 2. Initial acoustic conditions extracted from reference audio

[0084] 3. All symbols predicted before

[0085] This application can express the prediction process of the model as the following formula:

[0086]

[0087] a t+1 represents the next symbol predicted by the model at step t+1; θ AR represents the autoregressive Transformer decoder model; p represents the input phoneme sequence (text prompt); represents the initial acoustic conditions extracted from the reference audio; a1, a2, ..., a t Represents the sequence of symbols that the model has generated.

[0088] In the embodiment of the present application, since all data are discrete, the present application merges them into a continuous sequence without using any specific markers to distinguish different parts. For the symbols of the remaining levels, the present application trains a non-autoregressive (NAR) model to iteratively predict the symbol sequence of each layer. Each prediction step depends on the following two conditions:

[0089] 1. Phoneme sequence p: the input text information.

[0090] 2. The symbol sequence a generated by the previous level ＜j : That is, the symbols that the model has generated at the previous level.

[0091] In the embodiment of the present application, the prediction process of the NAR model can be expressed as the following formula:

[0092] a j =NAR(p,a ＜j )

[0093] a j : The symbol sequence predicted by the NAR model at the current level; NAR: non-autoregressive model; p: input phoneme sequence (text prompt); a ∫j : The symbol sequence generated by the previous level.

[0094] In an embodiment of the present application, a codec-based speech synthesis method is provided, comprising: receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes text prompt data and reference audio data; inputting the text prompt data into a phoneme conversion module for performing a phoneme conversion operation to obtain a phoneme sequence p; performing a feature extraction operation on the reference audio data to obtain an initial acoustic condition The phoneme sequence p and the initial acoustic condition The code is input into a codec-based speech synthesis model for coding prediction operation to obtain a coding prediction sequence; the coding prediction sequence is decoded to obtain the target synthesized audio. Compared with the prior art, the present application predicts multiple tags in each inference step of the autoregressive model through multiple prediction heads, thereby achieving the effect that the synthesis time decreases linearly with the increase in the number of tags.

[0095] In some optional implementations of the embodiments of the present application, the codec-based speech synthesis model is composed of an autoregressive module and a non-autoregressive module, which converts the phoneme sequence p and the initial acoustic condition The step of inputting into a codec-based speech synthesis model for coding prediction operation to obtain a coding prediction sequence specifically includes the following steps:

[0096] The phoneme sequence p and the initial acoustic condition Input to the autoregressive module for latent variable encoding operation to obtain the latent variable z 1:t , where the latent variable z 1:t It is expressed as:

[0097]

[0098] Among them, z 1:t represents the latent variable, representing the encoded information, a 1:t represents the previously generated symbol sequence, Denotes the initial acoustic condition p represents the phoneme sequence, θ AR represents the parameters of the autoregressive module;

[0099] According to the latent variable z 1:t And the non-autoregressive module performs multi-head prediction operations to obtain the encoded prediction sequence.

[0100] In the embodiment of the present application, the present application divides the autoregressive (AR) module into two main steps:

[0101] 1. Encoding step: First, all previously generated symbols are encoded into the latent space Among them, z 1:t represents the latent variable, representing the encoded information, a 1:T represents the previously generated symbol sequence, Denotes the initial acoustic condition p represents the phoneme sequence, θ AR Represents the parameters of the autoregressive module.

[0102] 2. Prediction step: Next, use this latent variable z 1:t To predict the next n consecutive symbols p(a 1+t:t+n |z 1:t ). This process is completed by n different attention heads, each of which is responsible for predicting a symbol.

[0103] In some optional implementations of the embodiments of the present application, the above-mentioned latent variable z 1:tThe steps of performing multi-head prediction operation with a non-autoregressive module to obtain a coding prediction sequence specifically include the following steps:

[0104] The multi-head prediction operation is performed according to the negative log-likelihood objective function, where the negative log-likelihood objective function It is expressed as:

[0105]

[0106] in, Represents the multi-head prediction negative log-likelihood loss function of the autoregressive module; Indicates that given a phoneme sequence p, acoustic conditions and the autoregressive module parameters θ AR In the case of 1:t+n probability.

[0107] In the embodiments of the present application, a new negative log-likelihood objective function is introduced for the entire multi-head AR module.

[0108] This application can express the new negative log-likelihood objective function as:

[0109]

[0110] in, Represents the multi-head prediction negative log-likelihood loss function of the autoregressive module; Indicates that given a phoneme sequence p, acoustic conditions and the autoregressive module parameters θ AR In the case of 1:t+n probability.

[0111] For better understanding, this application can decompose the loss function into multiple parts:

[0112]

[0113] here, It means that given the latent variable z 1:t and model parameters θ AR In the case of , predict the probability of the t+ith symbol.

[0114] By taking the logarithm of the probabilities of all n prediction steps and summing them, this application obtains the negative log-likelihood loss of the entire AR module.

[0115] 1. Coding stage:

[0116] The AR module first converts the previously generated symbol a 1:t Acoustic conditions Encoded into the latent space z1:t .

[0117] This step uses θ AR The parameterized model is completed.

[0118] 2. Prediction stage:

[0119] Using the latent variable z 1:t To predict the next n symbols.

[0120] The prediction of each symbol is handled by a different attention head, ensuring parallel processing.

[0121] In some optional implementations of the present application, the negative log-likelihood loss is calculated And update the model parameters through back propagation.

[0122] Through this multi-head prediction method, the AR module can effectively separate and parallelize encoding and prediction, thereby accelerating the decoding process. This method not only improves the efficiency of the model, but also enhances the expressiveness of the model through the multi-head mechanism, making the generated speech signal more natural and accurate.

[0123] In some optional implementations of the embodiments of the present application, the above-mentioned latent variable z 1:t The steps of performing multi-head prediction operation with a non-autoregressive module to obtain a coding prediction sequence specifically include the following steps:

[0124] The coding prediction sequence obtained by the multi-head prediction operation is optimized according to the heuristic decoding technology based on the Viterbi algorithm to obtain the optimized coding prediction sequence P(a i →a j ), where the optimized coding prediction sequence P(a i →a j ) is expressed as:

[0125] P(a i →a j )=Q(a i ,a j )·S t (a j )

[0126] Among them, Q(a i ,a j ) represents the symbol a i Immediately after the symbol a j The previous probability, where is the transfer matrix learned from the dataset, S t (a j ) means at time t, symbol a jThe probability of , which is obtained from the t forecast heads of the long forecast.

[0127] In the embodiment of the present application, in the multi-head prediction (MTP), the present application attempts to predict multiple symbols a at one time. 1+t:t+n , these predictions are based on the previously generated symbols a ≤t However, this approach will lead to context mismatches between predicted symbols. Specifically, except for the first symbol a 1+t In addition, other symbols a 2+t:t+n The prediction lacks complete knowledge of the previous symbols, which may lead to the accumulation of errors and ultimately affect the accuracy of the entire sequence.

[0128] In order to solve the above problems, this application proposes a heuristic decoding technology based on the Viterbi algorithm. The main goal of this technology is to use the top-k candidate symbols generated by multi-head prediction (MTP) to fill the context gap between consecutive predicted symbols. Among them:

[0129] -V: represents the symbol set of the first layer of the RVQ model. This application assumes that each symbol only depends on its previous symbol, so a Markov chain can be used to model the transition between symbols.

[0130] -Q(a i ,a j ): indicates the symbol a i Immediately after the symbol a j The previous probability, where is the transfer matrix learned from the dataset.

[0131] -S t (a i ): indicates that at time t, symbol a i The probability of , which is obtained from the t prediction heads of MTP.

[0132] Therefore, from the symbol a at time t-1 i The symbol a transferred to time t j The probability can be expressed as:

[0133] P(a i →a j )=Q(a i ,a j )·S t (a j )

[0134] In order to find the optimal symbol sequence A * ,This application applies a heuristic decoding technique based on the Viterbi algorithm.

[0135] In some optional implementations of the embodiments of the present application, the time complexity of directly implementing the above Viterbi algorithm is O(n·V 2 ), which is computationally very expensive. To alleviate this problem, the present application reduces the dimensions of the transfer matrix and the state matrix by selecting only relevant elements in each reasoning step. Specifically, the present application reduces the dimension of the transfer matrix from to, where is the total number of unique symbols with the top-k highest probability predictions in the n prediction heads of MTP. In practical applications, m is much smaller than V, which reduces the new computational complexity to, significantly reducing the computational overhead of each MTP reasoning.

[0136] Through this method, the present application not only solves the context mismatch problem in multi-head prediction, but also reduces the computational complexity by optimizing the Viterbi algorithm, making the technology more efficient and practical in practical applications.

[0137] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0138] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0139] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0140] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0141] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a speech synthesis device based on a codec, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0142] like Figure 3 As shown, the speech synthesis device 200 based on the codec of the embodiment of the present application includes:

[0143] The request receiving module 210 is used to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes text prompt data and reference audio data;

[0144] The phoneme conversion module 220 is used to input the text prompt data into the phoneme conversion module for performing a phoneme conversion operation to obtain a phoneme sequence p;

[0145] The feature extraction module 230 is used to perform feature extraction operations on the reference audio data to obtain initial acoustic conditions.

[0146] The coding prediction module 240 is used to convert the phoneme sequence p and the initial acoustic condition Input into a codec-based speech synthesis model to perform a coding prediction operation to obtain a coding prediction sequence;

[0147] The decoding module 250 is used to perform a decoding operation on the coded prediction sequence to obtain a target synthesized audio.

[0148] In an embodiment of the present application, a speech synthesis device 200 based on a codec is provided, comprising: a request receiving module 210, for receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes text prompt data and reference audio data; a phoneme conversion module 220, for inputting the text prompt data into a phoneme conversion module for performing a phoneme conversion operation to obtain a phoneme sequence p; a feature extraction module 230, for performing a feature extraction operation on the reference audio data to obtain an initial acoustic condition The coding prediction module 240 is used to convert the phoneme sequence p and the initial acoustic condition The code is input to the codec-based speech synthesis model for coding prediction operation to obtain a coding prediction sequence; the decoding module 250 is used to decode the coding prediction sequence to obtain the target synthesized audio. Compared with the prior art, the present application predicts multiple tags in each inference step of the autoregressive model through multiple prediction heads, thereby achieving the effect of linearly reducing the synthesis time as the number of tags increases.

[0149] In some optional implementations of the embodiments of the present application, the above-mentioned codec-based speech synthesis model is composed of an autoregressive module and a non-autoregressive module, and the coding prediction module includes:

[0150] The latent variable encoding submodule is used to transform the phoneme sequence p and the initial acoustic condition Input to the autoregressive module for latent variable encoding operation to obtain the latent variable z 1:t , where the latent variable z 1:t It is expressed as:

[0151]

[0152] Among them, z 1:t represents the latent variable, representing the encoded information, a 1:t represents the previously generated symbol sequence, Denotes the initial acoustic condition p represents the phoneme sequence, θ AR represents the parameters of the autoregressive module;

[0153] The multi-head prediction submodule is used to predict the 1:t And the non-autoregressive module performs multi-head prediction operations to obtain the encoded prediction sequence.

[0154] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of a computer device according to an embodiment of the present application.

[0155] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0156] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0157] The memory 310 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 310 can be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 can also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card, etc. Of course, the memory 310 can also include both the internal storage unit of the computer device 300 and its external storage device. In the embodiment of the present application, the memory 310 is generally used to store the operating system and various application software installed on the computer device 300, such as computer-readable instructions of the speech synthesis method based on the codec, etc. In addition, the memory 310 can also be used to temporarily store various data that have been output or are to be output.

[0158] The processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 320 is generally used to control the overall operation of the computer device 300. In the embodiment of the present application, the processor 320 is used to run the computer-readable instructions or process data stored in the memory 310, such as computer-readable instructions for running the codec-based speech synthesis method.

[0159] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0160] The computer device provided in the present application predicts multiple tags in each reasoning step of the autoregressive model through multiple prediction heads, thereby achieving the effect of linearly reducing the synthesis time as the number of tags increases.

[0161] The present application also provides another embodiment, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the codec-based speech synthesis method as described above.

[0162] The computer-readable storage medium provided in the present application predicts multiple tags in each reasoning step of the autoregressive model through multiple prediction heads, thereby achieving the effect that the synthesis time decreases linearly with the increase in the number of tags.

[0163] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0164] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A codec-based speech synthesis method, characterized in that: The steps include: Receiving a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes text prompt data and reference audio data; Inputting the text prompt data into a phoneme conversion module for phoneme conversion operation to obtain a phoneme sequence p; Perform feature extraction on the reference audio data to obtain initial acoustic conditions The phoneme sequence p and the initial acoustic condition Input into a codec-based speech synthesis model to perform a coding prediction operation to obtain a coding prediction sequence; The coding prediction sequence is decoded to obtain the target synthesized audio.

2. The codec-based speech synthesis method according to claim 1, characterized in that: The codec-based speech synthesis model is composed of an autoregressive module and a non-autoregressive module. The step of inputting into a codec-based speech synthesis model for coding prediction operation to obtain a coding prediction sequence specifically includes the following steps: The phoneme sequence p and the initial acoustic condition Input into the autoregressive module to perform latent variable encoding operation to obtain the latent variable z 1:t , where the latent variable z 1:t It is expressed as: Among them, z 1:t represents the latent variable, representing the encoded information, a 1:t represents the previously generated symbol sequence, represents the initial acoustic condition from p represents the phoneme sequence, θ AR represents the parameters of the autoregressive module; According to the latent variable z 1:t And the non-autoregressive module performs a multi-head prediction operation to obtain the coding prediction sequence.

3. The codec-based speech synthesis method according to claim 2, characterized in that: According to the latent variable z 1:t The non-autoregressive module performs a multi-head prediction operation to obtain the coding prediction sequence, which specifically includes the following steps: The multi-head prediction operation is performed according to a negative log-likelihood objective function, wherein the negative log-likelihood objective function It is expressed as: in, represents the multi-head prediction negative log-likelihood loss function of the autoregressive module; It means that given the phoneme sequence p and the acoustic condition and the autoregressive module parameters θ AR In the case of 1:t+n probability.

4. The codec-based speech synthesis method according to claim 2, characterized in that: According to the latent variable z 1:t The non-autoregressive module performs a multi-head prediction operation to obtain the coding prediction sequence, which specifically includes the following steps: The coding prediction sequence obtained by the multi-head prediction operation is optimized according to the heuristic decoding technology based on the Viterbi algorithm to obtain the optimized coding prediction sequence P(a i →a j ), wherein the optimized coding prediction sequence P(a i →a j ) is expressed as: P(a i →a j )=Q(a i ,a j )·S t (a j ) Among them, Q(a i ,a j ) represents the symbol a i Immediately after the symbol a j The previous probability, where is the transfer matrix learned from the dataset, S t (a j ) means at time t, symbol a j The probability of , which is obtained from the t prediction heads of the multi-head prediction.

5. The codec-based speech synthesis method according to claim 4, characterized in that: The dimensions of the transfer matrix can be obtained from Reduce to Where m≤k·n is the total number of unique symbols predicted by the top-k highest probability among the n prediction heads of the multi-head prediction.

6. The codec-based speech synthesis method according to claim 3, characterized in that: In the case of the latent variable z 1:t After the step of performing a multi-head prediction operation on the non-autoregressive module to obtain the coding prediction sequence, the following steps are also included: According to the negative log-likelihood objective function The calculation results of the autoregressive module parameters θ AR Perform parameter update operation.

7. A speech synthesis device based on a codec, characterized in that: include: A request receiving module, configured to receive a speech synthesis request sent by a user terminal, wherein the speech synthesis request includes text prompt data and reference audio data; A phoneme conversion module, used for inputting the text prompt data into the phoneme conversion module for performing a phoneme conversion operation to obtain a phoneme sequence p; A feature extraction module is used to perform feature extraction operations on the reference audio data to obtain initial acoustic conditions The coding prediction module is used to convert the phoneme sequence p and the initial acoustic condition Input into a codec-based speech synthesis model to perform a coding prediction operation to obtain a coding prediction sequence; The decoding module is used to perform a decoding operation on the coded prediction sequence to obtain a target synthesized audio.

8. The codec-based speech synthesis device according to claim 7, characterized in that: The codec-based speech synthesis model is composed of an autoregressive module and a non-autoregressive module, and the coding prediction module includes: A latent variable encoding submodule is used to convert the phoneme sequence p and the initial acoustic condition Input into the autoregressive module to perform latent variable encoding operation to obtain the latent variable z 1:r , where the latent variable z 1:t It is expressed as: Among them, z 1:t represents the latent variable, representing the encoded information, a 1:t represents the previously generated symbol sequence, represents the initial acoustic condition from p represents the phoneme sequence, θ AR represents the parameters of the autoregressive module; The multi-head prediction submodule is used to predict the 1:t And the non-autoregressive module performs a multi-head prediction operation to obtain the coding prediction sequence.

9. A computer device comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the codec-based speech synthesis method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the codec-based speech synthesis method according to any one of claims 1 to 6 are implemented.

Images