A method and apparatus for training an audio codec model

Abstract

This application discloses a training method and apparatus for an audio codec model. The main technical solution includes: acquiring training data comprising multiple audio samples; training the audio codec model using the training data; wherein the training includes: extracting the spectral representation of the audio samples; extracting the acoustic feature representation of the audio samples using a first audio encoder; mapping the acoustic feature representation to a latent space using a second audio encoder to obtain the latent space feature representation of the audio samples; masking the latent space feature representation to obtain the masked latent space feature representation; performing decoding processing based on the latent space feature representation using an audio decoder to obtain the reconstructed spectrum; and updating the model parameters of the audio codec model using the reconstructed spectrum and the spectral representation. This approach can improve the semantic understanding capability of the audio codec model, thereby improving the performance of speech generation tasks.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a training method and device of an audio codec model. BACKGROUND

[0002] With the continuous development of artificial intelligence, text-to-speech (TTS) and text-to-audio (TTA) technologies have emerged. Both of these technologies are essentially speech generation tasks, that is, converting received text into corresponding audio and feeding it back to the user. In the process of converting text to audio, an audio codec model is needed, but the performance of traditional audio codec models in semantic understanding is still lacking, which leads to poor performance of the speech generation task. SUMMARY

[0003] The present application provides a training method and device of an audio codec model to improve the semantic understanding ability of the audio codec model and thus improve the performance of the speech generation task.

[0004] The present application provides the following solutions:

[0005] According to a first aspect, a training method of an audio codec model is provided, the audio codec model comprising a first audio encoder, a second audio encoder and an audio decoder, and the method comprising:

[0006] obtaining training data comprising a plurality of audio samples;

[0007] training the audio codec model using the training data; wherein the training comprises: extracting a spectral representation of the audio sample, inputting the spectral representation into the first audio encoder; extracting an acoustic feature representation of the audio sample using the first audio encoder; mapping the acoustic feature representation into a latent space using the second audio encoder to obtain a latent space feature representation of the audio sample; performing mask processing on the latent space feature representation to obtain a masked latent space feature representation; performing decoding processing based on the latent space feature representation using the audio decoder to obtain a reconstructed spectrum; and updating model parameters of the audio codec model using the reconstructed spectrum and the spectral representation.

[0008] According to an implementable manner in the embodiments of the present application, the updating of the model parameters of the audio codec model using the reconstructed spectrum and the spectral representation comprises:

[0009] determining a value of a loss function using the reconstructed spectrum and the spectral representation;

[0010] update the model parameters of the audio codec model by using the value of the loss function;

[0011] The value of the loss function is determined by the values of at least two of the first loss function term, the second loss function term and the third loss function term; the value of the first loss function term is obtained from the difference between the reconstructed spectrum and the spectrum representation; the value of the second loss function term is obtained from the KL divergence between the distribution of the latent space feature representation and the standard normal distribution; and the value of the third loss function term is obtained from the probability that the discriminator distinguishes the reconstructed spectrum and the spectrum representation.

[0012] According to an implementable manner in the embodiments of the present application, the second audio encoder comprises a multi-layer fully connected layer.

[0013] The mapping of the acoustic feature representation into the latent space by using the second audio encoder to obtain the latent space feature representation comprises: predicting the distribution parameters corresponding to each time frame based on the acoustic feature representation by using the multi-layer fully connected layer, wherein the distribution parameters comprise mean and variance; performing reparameterization sampling based on the distribution parameters corresponding to each time frame respectively, and introducing Gaussian noise in the reparameterization sampling; and obtaining the latent space feature representation by using the latent space variables obtained from the reparameterization sampling.

[0014] According to an implementable manner in the embodiments of the present application, the mask processing of the latent space feature representation to obtain the masked latent space feature representation comprises:

[0015] Mask processing the latent space variables corresponding to at least one time frame in the latent space feature representation to obtain the masked latent space feature representation.

[0016] According to an implementable manner in the embodiments of the present application, the value of the second loss function term is obtained by adding a marginal constraint to the KL divergence between the distribution of the latent space feature representation and the standard normal distribution.

[0017] According to a second aspect, a speech synthesis method is provided, which comprises:

[0018] Obtaining a text to be synthesized;

[0019] Extracting a text feature representation of the text to be synthesized by using a text encoder;

[0020] Mapping the text feature representation into an acoustic feature representation by using an acoustic encoder;

[0021] Mapping the acoustic feature representation into a latent space by using a second audio encoder to obtain a latent space feature representation;

[0022] decode the latent space feature representation by using an audio decoder to obtain speech corresponding to the text to be synthesized.

[0023] The second audio encoder and the audio decoder are pre-trained by using the method in any one of the first aspects.

[0024] According to a third aspect, a device for training an audio codec model is provided, the audio codec model comprising a first audio encoder, a second audio encoder, and an audio decoder, the device comprising:

[0025] a sample obtaining unit configured to obtain training data comprising a plurality of audio samples;

[0026] a model training unit configured to train the audio codec model by using the training data; wherein the training comprises: extracting a spectral representation of the audio samples, inputting the spectral representation into the first audio encoder; extracting an acoustic feature representation of the audio samples by using the first audio encoder; mapping the acoustic feature representation into a latent space by using the second audio encoder to obtain a latent space feature representation of the audio samples; performing mask processing on the latent space feature representation to obtain a masked latent space feature representation; performing decoding processing based on the latent space feature representation by using the audio decoder to obtain a reconstructed spectrum; and updating model parameters of the audio codec model by using the reconstructed spectrum and the spectral representation.

[0027] According to a fourth aspect, a device for speech synthesis is provided, the device comprising:

[0028] a obtaining unit configured to obtain text to be synthesized;

[0029] a text encoding unit configured to extract a text feature representation of the text to be synthesized by using a text encoder;

[0030] an acoustic encoding unit configured to map the text feature representation into an acoustic feature representation by using an acoustic encoder;

[0031] a mapping unit configured to map the acoustic feature representation into a latent space by using a second audio encoder to obtain a latent space feature representation;

[0032] a decoding unit configured to decode the latent space feature representation by using an audio decoder to obtain speech corresponding to the text to be synthesized; wherein the second audio encoder and the audio decoder are pre-trained by using the method in any one of the first aspects.

[0033] According to a fifth aspect, a computer readable storage medium is provided, having stored thereon a computer program which, when executed by a processor, implements the steps of the method according to any one of the first and second aspects.

[0034] According to a sixth aspect, an electronic device is provided, comprising:

[0035] one or more processors; and

[0036] a memory associated with the one or more processors, the memory for storing program instructions which, when read and executed by the one or more processors, perform the steps of the method according to any one of the first and second aspects.

[0037] According to a seventh aspect, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method according to any one of the first and second aspects.

[0038] According to the embodiments provided in the present application, the following technical effects are disclosed:

[0039] 1) In the model training scheme provided in the present application, the audio codec model can be trained using training data comprising a plurality of audio samples. In the training process, the acoustic feature representation of the audio sample can be extracted using the first audio encoder. Then, the hidden space feature representation of the acoustic feature representation in the hidden space is obtained using the second audio encoder, and the hidden space feature representation is masked. Finally, the hidden space feature representation is decoded using the audio decoder to obtain the reconstructed spectrum. The model parameters of the audio codec model are updated using the reconstructed spectrum and the spectrum representation. In the present application, the hidden space feature representation is masked, which forces the first audio encoder to learn the context dependence of the audio sample, thereby improving the semantic understanding ability of the audio, and further improving the performance of the downstream speech generation task.

[0040] 2) In the present application, the loss function determined by the reconstructed spectrum and the spectrum representation is used to update the model parameters of the audio codec model, wherein the value of the loss function is determined by at least two of the first loss function item, the second loss function item and the third loss function item. In this way, when training the audio codec model, multiple loss function items such as reconstruction loss item, adversarial loss item and KL divergence loss can be used to update the model parameters. The basic structure of the audio is guaranteed not to be lost through the reconstruction loss item, the audio generated by the audio codec model is forced to be closer to the real audio in distribution through the adversarial loss, the natural feeling of the reconstructed audio in perception is improved, and the distribution continuity is constrained through the KL divergence loss, thereby improving the robustness of the audio codec model.

[0041] 3) The distribution parameters corresponding to each time frame are predicted through the full connection layer in the second audio encoder, and the distribution parameters are used for reparameterization sampling, so as to obtain the hidden space feature representation corresponding to the acoustic feature representation. That is, by introducing Gaussian noise, the audio coding and decoding model can be trained using a gradient-based optimization algorithm such as a stochastic gradient descent algorithm, accelerating the convergence of the model, improving the efficiency of the model training, and improving the stability of the model.

[0042] 4) When the hidden space feature representation is masked, the smallest unit of the mask is the hidden space variable corresponding to each time frame. In this way, the hidden space variable corresponding to each time frame can be forced to record the hidden space variables corresponding to the upper and lower time frames, so that the first audio encoder can learn the time sequence causal logic of the audio sample, and thus obtain more accurate acoustic feature representation.

[0043] 5) The marginal constraint is added to the second loss function term to dynamically regulate the second loss function term based on the KL divergence. Intervention is only performed when the KL divergence exceeds a reasonable range, which ensures the reasonableness of the distribution of the hidden space feature representation and avoids excessive constraints.

[0044] 6) Since the audio coding and decoding model trained by the training method provided by the present application has better semantic understanding ability, the speech synthesized based on the trained audio coding and decoding model is more accurate, and the quality of the generated speech is improved.

[0045] Of course, implementing any product of the present application does not necessarily require all the advantages described above. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0047] Figure 1 The system architecture diagram to which the embodiments of the present application are applicable.

[0048] Figure 2 The flowchart of the training method of the audio coding and decoding model provided by the embodiments of the present application.

[0049] Figure 3 The schematic diagram of the hidden space feature representation provided by the embodiments of the present application.

[0050] Figure 4 The schematic diagram of the two masking methods provided by the embodiments of the present application.

[0051] Figure 5 A schematic diagram of an overall training process provided for embodiments of the present application.

[0052] Figure 6 A flowchart of a speech synthesis method provided for embodiments of the present application.

[0053] Figure 7 A schematic block diagram of a training apparatus of an audio codec model provided for embodiments of the present application.

[0054] Figure 8 A schematic block diagram of a speech synthesis apparatus provided for embodiments of the present application.

[0055] Figure 9 A schematic block diagram of an electronic device provided for embodiments of the present application. DETAILED DESCRIPTION

[0056] The technical solutions in embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art belong to the scope of protection of the present application.

[0057] The terms used in the embodiments of the present application are merely for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms “a”, “an” and “the” used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0058] It should be understood that the term “and / or” used herein is merely to describe an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character “ / ” herein generally represents an “or” relationship between the front and rear associated objects.

[0059] Depending on the context, the word “if” as used herein can be interpreted as “when” or “upon” or “in response to determining” or “in response to detecting”. Similarly, depending on the context, the phrase “if it is determined” or “if (a stated condition or event) is detected” can be interpreted as “when it is determined” or “in response to determining” or “when (a stated condition or event) is detected” or “in response to detecting (a stated condition or event)”.

[0060] Currently, there are some technologies that can be applied to speech synthesis tasks, such as generating speech corresponding to text sequences using diffusion models based on text sequences. However, the speech generated by this method often suffers from semantic loss, lack of continuity, and low quality.

[0061] Figure 1 The system architecture diagram applicable to this application is as follows: Figure 1 As shown, the system architecture may include: user equipment, a training device for the audio codec model located on the server side, and a speech synthesis device.

[0062] The user equipment and the server can communicate with each other. The user equipment and the server can be connected directly or indirectly through wired or wireless communication, which is not limited herein.

[0063] User devices can include, but are not limited to, smart mobile terminals, smart home devices, wearable devices, and PCs (Personal Computers). Smart mobile devices can include mobile phones, tablets, laptops, PDAs (Personal Digital Assistants), and connected cars. Smart home devices can include smart TVs, smart refrigerators, and so on. Wearable devices can include smartwatches, smart glasses, virtual reality devices, augmented reality devices, and mixed reality devices (i.e., devices that support both virtual and augmented reality).

[0064] A server can be a standalone server, a server cluster, or a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a hosting product within the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Servers (VPS) services, such as high management difficulty and weak service scalability.

[0065] Before performing a speech synthesis task, the training device for the audio codec model can be trained using the method provided in the embodiments of this application, which includes a pre-trained second audio encoder and audio decoder.

[0066] Users can input text via their user devices, which then send the text to be synthesized to a speech synthesis device on the server via the network. The speech synthesis device uses a pre-trained second audio encoder and audio decoder to generate the speech corresponding to the text to be synthesized, and returns the speech to the user device via the network. The user device then plays the received speech back to the user.

[0067] Apart from Figure 1In addition to the architecture shown, audio codec models can also be trained and / or speech generated by computer terminal devices with strong computing power using the methods provided in the embodiments of this application.

[0068] It should be understood that Figure 1 The number of user devices, audio codec model training devices, and speech synthesis devices shown in the diagram is merely illustrative. Depending on implementation needs, any number of user devices, audio codec model training devices, and speech synthesis devices can be included.

[0069] In view of this, this application provides a new approach to training audio codec models. Figure 2 This is a flowchart illustrating the training method for the audio codec model provided in this application embodiment. This method can be performed by… Figure 1 The training device for the audio codec model in the system shown is executed. For example... Figure 2 As shown, the method may include the following steps:

[0070] Step 201: Obtain training data including multiple audio samples.

[0071] Step 202: Train the audio codec model using the training data; wherein, the training includes: extracting the spectral representation of the audio samples and inputting the spectral representation into the first audio encoder; extracting the acoustic feature representation of the audio samples using the first audio encoder; mapping the acoustic feature representation into the latent space using the second audio encoder to obtain the latent space feature representation of the audio samples; masking the latent space feature representation to obtain the masked latent space feature representation; decoding the audio samples based on the latent space feature representation using the audio decoder to obtain the reconstructed spectrum; and updating the model parameters of the audio codec model using the reconstructed spectrum and the spectral representation.

[0072] As can be seen from the above process, the model training scheme provided in this application can utilize training data including multiple audio samples to train the audio codec model. During training, the acoustic feature representation of the audio samples is first extracted using a first audio encoder. Then, the latent space feature representation of the acoustic feature representation is obtained using a second audio encoder, and this latent space feature representation is masked. Finally, the latent space feature representation is decoded using an audio decoder to obtain the reconstructed spectrum. The model parameters of the audio codec model are updated using the reconstructed spectrum and the spectral representation. By masking the latent space feature representation, this application forces the first audio encoder to learn the contextual dependencies of the audio samples, thereby improving the semantic understanding capability of the audio and thus improving the performance of downstream speech generation tasks.

[0073] The following describes in detail step 202 of the above process and the effects that can be further produced, with reference to the embodiments. It should be noted that the terms "first" and "second" involved in this disclosure do not have limitations in terms of size, order, or quantity, but are only used to distinguish them in name. For example, "first audio encoder" and "second audio encoder" are used to distinguish two audio encoders.

[0074] In the embodiments of this application, when training an audio codec model using training data including multiple audio samples, the audio samples can first be converted into a spectral representation. The audio samples can refer to speech audio or non-speech audio (such as environmental sound effects, music, scene sounds, etc.).

[0075] The spectral representation here can refer to the Mel-frequency spectrum representation. As one embodiment, the audio sample (i.e., the audio signal) is divided into multiple fixed-length time frames, for example, a frame length of 25ms. To ensure continuity between time frames, there is some overlap, such as a frame shift of 10ms. A window function (such as a Hanning window) is applied to the audio signal corresponding to each time frame to reduce spectral leakage and ensure a smooth transition of the audio signal at the boundaries of the time frames. Then, a Fourier transform is performed on the audio signal corresponding to each windowed time frame, converting the time-domain signal into a frequency-domain signal, thus obtaining the spectrum corresponding to each time frame. Next, a set of Mel filters (typically 64 filters, i.e., 64 Mel bands) is designed. The spectrum of each time frame obtained through the Fourier transform is projected onto the Mel filter bank to convert the spectrum from a linear frequency scale to a Mel frequency scale, which better matches the perceptual characteristics of the human ear. The logarithm of the projection result of the Mel filter bank is taken to obtain the logarithmic Mel spectrum, which is the spectral representation of the audio sample.

[0076] In addition, after obtaining the spectral representation of the audio samples, the spectral representation is usually normalized to the [-1,1] interval along the frequency axis to eliminate dimensional differences and accelerate training convergence.

[0077] Of course, the above-mentioned spectral representation can also be other spectral representations, such as sampling point representation, Fourier transform-extracted spectral energy distribution representation, and short-time Fourier transform representation. This application does not make any specific limitations. Among them, sampling point representation discretizes the continuous audio signal at a fixed frequency (such as 16kHz, 44.1kHz) to obtain sample values, and then maps the sample values ​​to numbers to generate a one-dimensional array that directly reflects the changes of the audio signal over time; Fourier transform-extracted spectral energy distribution representation performs a Fourier transform on each short frame (such as 25ms) of the audio signal to obtain the energy distribution of each frequency component in the frequency domain; short-time Fourier transform representation performs a Fourier transform on each frame after the audio signal is divided into frames to generate a two-dimensional time-frequency matrix, while retaining time and frequency information.

[0078] It should be noted that, in order to facilitate the subsequent calculation of the difference between the spectral representation and the reconstructed spectrum of the audio sample, the same spectral representation method should be used for both.

[0079] Next, the extracted spectral representation is input into the first audio encoder, which then extracts the acoustic feature representation of the audio samples based on the spectral representation. The network structure of the first audio encoder can be any network structure that can achieve feature extraction, such as Transformer network, Convolutional Neural Network (CNN), Gated Recurrent Unit (GRU), etc.

[0080] In a preferred embodiment, the first audio encoder may include causal convolutional layers and a Transformer layer. This application sequentially inputs the spectral representation into each causal convolutional layer to obtain local spectral feature representations of the spectral representation. Then, these local spectral feature representations are input into the Transformer layer so that the Transformer layer can capture long-term temporal dependencies in the audio samples, resulting in the acoustic feature representation output by the Transformer layer. The acoustic feature representation can be a temporal feature matrix H. (H is) The acoustic feature representation is a two-dimensional matrix, where T represents the number of time frames and d represents the feature dimension (d can be equal to 40). This acoustic feature representation includes both local spectral information and global temporal context information, allowing for a more accurate description of the features contained in the audio samples.

[0081] Furthermore, by using a second audio encoder, the acoustic feature representation is mapped to the latent space to obtain the latent space feature representation of the audio sample. For example, this application can map the acoustic feature representation to the latent space of a normalized flow model. The normalized flow model can accurately calculate the probability density of the acoustic feature representation in the latent space through an invertible transformation. Moreover, since the transformation is invertible, the acoustic feature representation can be accurately converted between the data space and the latent space, which is beneficial for in-depth analysis and processing of audio data.

[0082] In a preferred embodiment, the second audio encoder can be implemented based on a variational autoencoder (VAE). This second audio encoder includes multiple fully connected layers. Using these fully connected layers, based on acoustic feature representations, it predicts the distribution parameters corresponding to each time frame. These distribution parameters include the mean and variance. If two fully connected layers exist (hereinafter referred to as the first fully connected layer and the second fully connected layer), the acoustic feature representations can be input into the first fully connected layer to predict the mean corresponding to each time frame, and the acoustic feature representations can be input into the second fully connected layer to predict the variance corresponding to each time frame.

[0083] It should be noted that in each training round, the training data input into the audio codec model includes multiple audio samples (e.g., 32 audio samples). Each audio sample is converted into a spectral representation, and the acoustic feature representation of each audio sample is obtained. Then, each acoustic feature representation is input into the second audio encoder. At this time, the feature dimension d and the number of time frames T of each acoustic feature representation input into the second audio encoder should be consistent. If they are inconsistent, the shorter acoustic feature representation can be padded. Predicting the mean and variance corresponding to each time frame actually means, for a certain time frame, predicting the mean and variance of the feature representation of each acoustic feature representation at that time frame.

[0084] Then, reparameterization sampling is performed based on the distribution parameters corresponding to each time frame. Gaussian noise is introduced into the reparameterization sampling, and the latent space variables obtained from the reparameterization sampling are used to obtain the latent space feature representation. For example... Figure 3 As shown, Figure 3 This diagram illustrates a latent space feature representation. The area enclosed by the dashed box represents the latent space variables corresponding to a specific time frame. In other words, for each time frame, the latent space variables for that time frame are obtained based on the mean and variance. Based on these latent space variables, the latent space feature representation is derived. The process of obtaining the latent space variables based on the mean and variance can be expressed by the following formula:

[0085]

[0086] Where z is the latent space variable corresponding to each time frame. Standard deviation (S) ),and , Represents the mean, and , and Let d be a k-dimensional vector, where k is the feature dimension of the latent space. k and d should be the same, such as d=40, then k=40. Gaussian noise is used to ensure gradient backtracking. From A random vector sampled from (i.e., a standard normal distribution). This indicates element-wise multiplication.

[0087] After obtaining the latent space feature representation corresponding to the acoustic feature representation, the latent space feature representation is masked. The purpose of this masking is to force the first audio encoder to learn the contextual dependencies of each time frame in the audio samples. As an optional implementation method, such as... Figure 4 As shown in (a), the latent space feature representation is subjected to element-level masking, and the smallest unit of masking is the element. Figure 4 In (a), each square represents an element in the latent space feature representation. Black squares represent elements that have not been masked, and white squares represent elements that have been masked.

[0088] As a preferred embodiment, the latent space feature representation is subjected to temporal frame-level masking, where the smallest unit of masking is all elements corresponding to a single temporal frame. For example... Figure 4 As shown in (b) in the figure, Figure 4In (b) of the diagram, each square represents an element in the latent space feature representation. Black squares represent elements that have not been masked, and white squares represent elements that have been masked. Specifically, based on a preset masking rate, the latent space variables corresponding to at least one time frame in the latent space feature representation can be randomly masked. The masking rate in this application can be set to 40%~60%. When masking, short-span masks and long-span masks can be performed. For example, given a latent space feature representation that includes 100 time frames, the feature representation of frame 20 can be masked (short-span mask), and the feature representations of frames 25~33 can also be masked (long-span mask). In practical applications, short-span masks are more common, while long-span masks are less common. For the feature representations that have been masked, this application can use learnable mask markers [MASK] to replace them, while retaining the position information of these masked feature representations. In subsequent learning, the audio decoder can know which positions have been masked, thereby making targeted predictions. In this way, audio codec models can learn the temporal causal logic within audio samples, such as the temporal order and causal relationships of birdsong in an audio sample. This helps audio codec models better understand the temporal structure and semantic information of audio, improving their ability to understand and model the content of audio samples.

[0089] Furthermore, an audio decoder is used to perform decoding based on the latent space feature representation after masking to obtain the reconstructed spectrum. The audio decoder can be a fully connected layer-based decoder, a recurrent neural network-based decoder, a Transformer-based decoder, a diffusion model-based decoder, etc., and this application does not impose any restrictions.

[0090] As a preferred embodiment, the audio decoder may include multiple transposed convolutional layers, which are used to upsample the latent space feature representation. Transposed convolution (also known as deconvolution) is a method that can increase data dimensionality while preserving spatial relationships. By performing multiple transposed convolution operations, the spatial dimension of the data is gradually increased, allowing the detailed information of the data to be recovered.

[0091] When the output reconstructed spectrum is the reconstructed Mel spectrum, the output reconstructed spectrum can be expressed as: (F is the number of megapixels, which can be equal to 128, and T is the number of time frames).

[0092] When training an audio codec model, the model parameters can be updated using the reconstructed spectrum and the spectral representation of the audio samples.

[0093] For example, the value of the loss function can be determined by using the reconstructed spectrum and the spectral representation of the audio samples, and the model parameters of the audio codec model can be updated using the value of the loss function.

[0094] As one feasible implementation, the value of the loss function can be determined by utilizing the difference between the reconstructed spectrum and the spectral representation of the audio sample.

[0095] However, the model trained in this way still needs improvement in terms of audio naturalness and robustness in audio generation tasks. Therefore, this application provides a preferred embodiment in which the value of the loss function is determined by the values ​​of at least two of the first, second, and third loss function terms. The following sections will describe these three loss function terms in detail.

[0096] First, the first loss function term is essentially the reconstruction loss, and its value is obtained from the difference between the reconstructed spectrum and the spectral representation of the audio sample. If the spectral representation is a Mel-spectral representation, the difference between the reconstructed Mel-spectrum and the Mel-spectral representation of the audio sample can be expressed by the formula:

[0097]

[0098] in, The Mel spectrum representation of the audio sample in the th... i The Mel frequency band, the first j The value of each time frame. Indicates the reconstruction of the Mel spectrum in the 1st century. i The Mel frequency band, the first j The value of each time frame. It is the normalization factor, F is the total number of Mel bands, and T is the total number of time frames.

[0099] Secondly, for the second loss function term, the value of the second loss function term includes the value of the second sub-loss function term corresponding to each time frame. That is, for each time frame, there is a corresponding second sub-loss function term. Based on the value of the second sub-loss function term corresponding to each time frame, the value of the second loss function term is obtained. For example, the sum of the values ​​of each second sub-loss function term is taken as the value of the second loss function term. Another example is that the average value of each second sub-loss function term is taken as the value of the second loss function term.

[0100] As one feasible implementation, the value of the second loss function term can be obtained from the KL divergence between the latent space feature representation distribution and the standard normal distribution. Specifically, the value of the second sub-loss function term for each time frame can be obtained based on the KL divergence between the latent space variables and the standard normal distribution for each time frame. The KL divergence (i.e., the second sub-loss function term) between the latent space variables and the standard normal distribution for each time frame can be expressed by the following formula:

[0101]

[0102] in, The mean, For variance, It follows a standard normal distribution. For the distribution of latent space variables, This represents the KL divergence between the distribution of the latent space variable and the standard normal distribution.

[0103] In practical applications, it is not only necessary to make the distribution of latent space variables approximate the standard normal distribution, but also to control the magnitude of the KL divergence. This is to prevent the latent space from losing diversity due to excessively small KL divergence, leading to a decrease in generative ability, or to prevent the latent space variables from deviating too much from the standard normal distribution, resulting in gradient explosion or vanishing, and consequently, training instability. Therefore, the value of the second sub-loss function term in this application can also be obtained by adding a marginal constraint to the KL divergence between the distribution of latent space variables and the standard normal distribution, which can be expressed by the formula:

[0104]

[0105] in, The mean, For variance, It follows a standard normal distribution. For the distribution of latent space variables, The KL divergence represents the distribution of the latent space variable relative to the standard normal distribution. As a marginal constraint, This indicates that when the KL divergence is greater than At that time, the second sub-loss function term is the KL divergence and The difference, if the KL divergence is less than or equal to At that time, the second sub-loss function term is 0.

[0106] In this way, marginal constraints are used. This serves as a reasonable boundary for the KL divergence, preventing it from becoming excessively large. When the KL divergence is greater than... The second sub-loss function term is activated only at that time. (Greater than 0) forces the audio codec model to adjust its parameters, making the distribution of latent space variables closer to a standard normal distribution, thus avoiding training collapse caused by the unbounded growth of KL divergence. At the same time, a moderately small KL divergence is allowed, i.e., when the KL divergence is less than or equal to... When the second sub-loss function term is 0, the audio codec model will not be penalized for having too small a KL divergence during training, thus giving the audio codec model a certain degree of training freedom.

[0107] Finally, the third loss function term is essentially an adversarial loss, the value of which is obtained by the probability of an additional discriminator distinguishing between the reconstructed spectrum and the spectral representation of the audio sample.

[0108] Specifically, the third loss function term may include a discriminant loss function term and a generation loss function term. The generation loss function term is obtained by the probability that the discriminator determines the reconstructed spectrum to be the true spectrum, and the discriminant loss function term is obtained by the probability that the discriminator determines the spectral representation of the audio sample to be the true spectrum and the probability that the reconstructed spectrum is the true spectrum. For example, the third loss function term may refer to the sum of the generation loss function term and the discriminant loss function term.

[0109] The generation loss function term can be expressed by the formula:

[0110]

[0111] in, E[] represents the probability that the discriminator will classify the reconstructed spectrum as the true spectrum. E[] is used to calculate the expectation. In actual training, a batch of audio samples is typically used to update the model parameters; E[] represents the probability of this batch of audio samples being correct. Averaging is performed to ensure that the generation loss function term more stably reflects the performance of the audio codec model on this batch of audio samples, avoiding excessive impact on the training process due to anomalies in a single audio sample. In this application, the audio codec model is trained with the goal of minimizing the generation loss function term. That is, for the training of the generator (i.e., the first audio encoder, the second audio encoder, and the audio decoder), this application aims to make the probability of the generated reconstructed spectrum discriminator identifying it as a real spectrum as close to 1 as possible.

[0112] The discriminant loss function term can be expressed by the formula:

[0113]

[0114] in, This indicates the probability that the discriminator will classify the spectral representation of an audio sample as the true spectral representation. E[] represents the probability that the discriminator will identify the reconstructed spectrum as the real spectrum, and E[] is used to calculate the expectation. In this application, the audio codec model is trained with the goal of minimizing the discriminant loss function term. That is, for the training of the discriminator, this application hopes that the probability of the generated reconstructed spectrum being identified as the real spectrum by the discriminator is as close to 0 as possible, and the probability of the spectrum representation of the audio sample being identified as the real spectrum by the discriminator is as close to 1 as possible.

[0115] For a simple example, in a training round, if the training data includes 32 audio samples, and based on these 32 audio samples, 32 reconstructed spectra are obtained, and these 32 reconstructed spectra are input into the discriminator, the discriminator will identify 12 of these reconstructed spectra as the true spectra. Given a frequency distribution of 12 / 32, and inputting the spectral representations of 32 audio samples into the discriminator, the discriminator identifies 25 of these audio samples as the true spectrum. It is 25 / 32.

[0116] It should be noted that the discriminator can be used to determine whether the input spectrum is a true spectrum, that is, whether the input spectrum comes from the real data distribution (i.e., the spectral representation of the audio samples) or from the generated data distribution (i.e., the reconstructed spectrum). The probability that the discriminator judges the input spectrum to come from the real data distribution is taken as the probability that the input spectrum is a true spectrum.

[0117] In a preferred embodiment, the discriminator added in this application is a multi-scale discriminator, which may include a first discriminator and a second discriminator. When determining whether the input spectrum is a true spectrum, the first discriminator and the second discriminator can output a first discrimination result and a second discrimination result, respectively. Only when both the first discrimination result and the second discrimination result indicate that the input spectrum is a true spectrum can it be determined that the input spectrum is a true spectrum.

[0118] The first discriminator is used to determine whether the spectrum corresponding to each time frame of the input spectrum is a real spectrum (single frame level judgment), that is, to detect the local authenticity of the input spectrum. The second discriminator is used to determine whether the spectrum corresponding to consecutive frames of the input spectrum is a real spectrum (segment level judgment), that is, to evaluate the overall coherence of the input spectrum. Here, a consecutive frame consists of multiple consecutive time frames, and the duration of the consecutive frames is greater than or equal to a preset threshold (such as 200ms).

[0119] For example, for a reconstructed spectrum of 100 frames, for each time frame, a first discriminator is used to determine the discriminant result for each time frame (i.e., 100 discriminant results). Based on these 100 discriminant results, a first discriminant result is obtained. Alternatively, the 100 time frames can be divided into 10 consecutive frames (the duration of each consecutive frame is greater than a preset threshold). Each consecutive frame includes 10 consecutive time frames, and a second discriminator is used to determine the discriminant result for each consecutive frame (i.e., 10 discriminant results). Based on these 10 discriminant results, a second discriminant result is obtained. Only when both the first and second discriminant results indicate that the reconstructed spectrum is the true spectrum can it be determined that the reconstructed spectrum is the true spectrum.

[0120] In the embodiments of this application, the value of the loss function is determined by the values ​​of at least two of the first loss function term, the second loss function term, and the third loss function term. For example, the value of the loss function may include the value of the first loss function term and the value of the second loss function term. Or, for another example, the value of the loss function may include the values ​​of the first loss function term and the third loss function term.

[0121] In a preferred embodiment, the loss function includes the values ​​of a first loss function term, a second loss function term, and a third loss function term, which can be expressed by the formula:

[0122]

[0123] in, For loss function, For the first loss function term, For the second loss function term, , For the third loss function term, , , , The weight parameter is used to adjust the weight of different loss function terms in the loss function.

[0124] It should be noted that, as a preferred embodiment, the optimizer used in this application during model training can be AdamW, with a learning rate of 3e-4 and a weight decay of 1e-3. The batch size of a batch of audio samples can be 32. Additionally, mixed-precision training (a technique combining single-precision (FP32) and half-precision (FP16) floating-point calculations) can be used to accelerate the training process of the audio codec model.

[0125] To provide a comprehensive explanation of the model training process, this application provides a complete example, such as... Figure 5 As shown.

[0126] fromFigure 5 As can be seen, the audio samples are first converted into a spectral representation, which is then input into the first audio encoder to obtain an acoustic feature representation. Using the second audio encoder, this acoustic feature representation is mapped into the latent space to obtain a latent space feature representation. This latent space feature representation is then masked to obtain a masked latent space feature representation. Finally, the audio decoder decodes this masked latent space feature representation to obtain the reconstructed spectrum. Based on the values ​​of the first loss function term, the second loss function term, the generation loss function term, and the discriminant loss function term, the model parameters of the audio encoding / decoding model, including the first audio encoder, the second audio encoder, the audio decoder, and the discriminator, are updated. Specifically, the value of the first loss function term is obtained from the difference between the reconstructed spectrum and the spectral representation; the value of the second loss function term is obtained from the KL divergence between the distribution of the latent space feature representation and the standard normal distribution; the generation loss function term is obtained from the probability that the discriminator determines the reconstructed spectrum to be the true spectrum; and the discriminant loss function term is obtained from the probability that the discriminator determines the spectral representation of the audio sample to be the true spectrum and the probability that the discriminator determines the reconstructed spectrum to be the true spectrum.

[0127] Figure 6 This is a flowchart illustrating the training method for the audio codec model provided in this application embodiment. This method can be performed by… Figure 1 The speech synthesis device in the system shown performs the operation. For example... Figure 6 As shown, the method may include the following steps:

[0128] Step 601: Obtain the text to be synthesized.

[0129] Step 602: Use a text encoder to extract the text feature representation of the text to be synthesized.

[0130] Step 603: Use an acoustic encoder to map the text feature representation to an acoustic feature representation.

[0131] Step 604: Using the second audio encoder, the acoustic feature representation is mapped to the latent space to obtain the latent space feature representation.

[0132] Step 605: Use the audio decoder to decode the latent space feature representation to obtain the speech corresponding to the text to be synthesized. The second audio encoder and audio decoder are pre-trained using the model training method described above.

[0133] In this embodiment, the second audio encoder and audio decoder in the trained audio codec model can be used to synthesize speech. Specifically, the text to be synthesized is obtained, the text feature representation of the text to be synthesized is extracted using the text encoder, the text feature representation is converted into an acoustic feature representation using the acoustic encoder, the acoustic feature representation is mapped into a latent space using the trained second audio encoder to obtain the latent space feature representation of the acoustic feature representation, and the latent space feature representation is decoded using the audio decoder to obtain the spectral representation corresponding to the text to be synthesized, and the spectral representation is restored to speech.

[0134] Furthermore, as an achievable embodiment, if the speech synthesis model for completing the speech synthesis task includes an encoder and a decoder, this application can directly train the encoder of the speech synthesis model using a first audio encoder. Specifically, multiple training sample pairs are obtained, each training sample pair including a text sample and the corresponding speech (hereinafter referred to as a speech sample). This application can input the speech sample into the first audio encoder to obtain the acoustic feature representation corresponding to the speech sample, and input the text sample into the encoder of the speech synthesis model to obtain the predicted feature representation corresponding to the text sample. The encoder of the speech synthesis model is trained with minimizing the difference between the predicted feature representation and the acoustic feature representation corresponding to the speech sample as the optimization objective. Of course, the audio encoding and decoding model of this application can also be used for audio reconstruction tasks, etc.

[0135] Therefore, the audio codec model trained in this application includes various model parts, each of which can perform a task as an independent entity or as a whole. This application does not limit the application scenarios of the audio codec model.

[0136] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0137] According to another embodiment, a training apparatus for an audio codec model is provided. Figure 7 A schematic block diagram of a training apparatus for the audio codec model according to one embodiment is shown, the apparatus being disposed in Figure 1 The server side in the illustrated architecture. For example... Figure 7 As shown, the audio codec model includes a first audio encoder, a second audio encoder, and an audio decoder. The device 700 includes:

[0138] The sample acquisition unit 701 is configured to acquire training data including multiple audio samples.

[0139] The model training unit 702 is configured to train an audio codec model using training data. The training includes: extracting the spectral representation of audio samples and inputting the spectral representation into a first audio encoder; extracting the acoustic feature representation of audio samples using the first audio encoder; mapping the acoustic feature representation into the latent space using a second audio encoder to obtain the latent space feature representation of the audio samples; masking the latent space feature representation to obtain the masked latent space feature representation; performing decoding based on the latent space feature representation using an audio decoder to obtain the reconstructed spectrum; and updating the model parameters of the audio codec model using the reconstructed spectrum and the spectral representation.

[0140] As one possible implementation, when updating the model parameters of the audio codec model using the reconstructed spectrum and spectral representation, the model training unit 702 can be configured to: determine the value of the loss function using the reconstructed spectrum and spectral representation; update the model parameters of the audio codec model using the value of the loss function; wherein the value of the loss function is determined by the values ​​of at least two of the first loss function term, the second loss function term, and the third loss function term; the value of the first loss function term is obtained by the difference between the reconstructed spectrum and the spectral representation; the value of the second loss function term is obtained by the KL divergence between the distribution of the latent space feature representation and the standard normal distribution; and the value of the third loss function term is obtained by the probability that the discriminator distinguishes between the reconstructed spectrum and the spectral representation.

[0141] As one possible implementation, the second audio encoder includes multiple fully connected layers.

[0142] When the model training unit 702 uses the second audio encoder to map the acoustic feature representation to the latent space to obtain the latent space feature representation, it can be configured as follows: using multiple fully connected layers to predict the distribution parameters corresponding to each time frame based on the acoustic feature representation, the distribution parameters including mean and variance; performing reparameterization sampling based on the distribution parameters corresponding to each time frame, introducing Gaussian noise in the reparameterization sampling; and using the latent space variables obtained by the reparameterization sampling to obtain the latent space feature representation.

[0143] As one possible approach, when the model training unit 702 performs masking processing on the latent space feature representation to obtain the masked latent space feature representation, it can be configured to: perform masking processing on the latent space variables corresponding to at least one time frame in the latent space feature representation to obtain the masked latent space feature representation.

[0144] As one possible approach, the value of the second loss function term is obtained by adding a marginal constraint to the KL divergence between the latent space feature representation distribution and the standard normal distribution.

[0145] According to another embodiment, a speech synthesis device is provided. Figure 8 A schematic block diagram of the speech synthesis apparatus according to one embodiment is shown, the apparatus being disposed in Figure 1 The server side in the illustrated architecture. For example... Figure 8 As shown, the device 800 includes:

[0146] Acquisition unit 801 is configured to acquire the text to be synthesized;

[0147] The text encoding unit 802 is configured to extract text feature representations of the text to be synthesized using a text encoder;

[0148] The acoustic coding unit 803 is configured to map text feature representations to acoustic feature representations using an acoustic encoder;

[0149] The mapping unit 804 is configured to use the second audio encoder to map the acoustic feature representation to the latent space to obtain the latent space feature representation;

[0150] The decoding unit 805 is configured to decode the latent space feature representation using the audio decoder to obtain the speech corresponding to the text to be synthesized; wherein the second audio encoder and the audio decoder are pre-trained using any of the above model training methods.

[0151] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0153] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0154] And an electronic device, comprising:

[0155] One or more processors; and

[0156] A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0157] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0158] in, Figure 9 An exemplary architecture of an electronic device is shown, which may include a processor 910, a video display adapter 911, a disk drive 912, an input / output interface 913, a network interface 914, and a memory 920. The processor 910, video display adapter 911, disk drive 912, input / output interface 913, network interface 914, and memory 920 can communicate with each other via a communication bus 930.

[0159] The processor 910 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs in order to implement the technical solution provided in this application.

[0160] The memory 920 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 920 can store the operating system 921 for controlling the operation of the electronic device 900, and the basic input / output system (BIOS) 922 for controlling the low-level operations of the electronic device 900. Additionally, it can store a web browser 923, a data storage management system 924, and an audio codec model training device 700 or a speech synthesis device 800, etc. The aforementioned audio codec model training device 700 or speech synthesis device 800 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 920 and executed by the processor 910.

[0161] Input / output interface 913 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0162] Network interface 914 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0163] Bus 930 includes a pathway for transmitting information between various components of the device, such as processor 910, video display adapter 911, disk drive 912, input / output interface 913, network interface 914, and memory 920.

[0164] It should be noted that although the above-described device only shows the processor 910, video display adapter 911, disk drive 912, input / output interface 913, network interface 914, memory 920, bus 930, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0165] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0166] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A training method for an audio codec model, characterized in that, The audio encoding / decoding model includes a first audio encoder, a second audio encoder, and an audio decoder; the method includes: Acquire training data including multiple audio samples; The audio codec model is trained using the training data; wherein the training includes: extracting the spectral representation of the audio sample and inputting the spectral representation into the first audio encoder; extracting the acoustic feature representation of the audio sample using the first audio encoder; mapping the acoustic feature representation into the latent space using the second audio encoder to obtain the latent space feature representation of the audio sample; performing masking processing on the latent space variables corresponding to at least one time frame in the latent space feature representation to obtain the masked latent space feature representation, the masking processing including short-span masking and long-span masking, wherein the masking time span of the short-span masking is shorter than that of the long-span masking, and the short-span masking has more cases than the long-span masking; performing decoding processing using the audio decoder based on the masked latent space feature representation to obtain the reconstructed spectrum; and updating the model parameters of the audio codec model using the reconstructed spectrum and the spectral representation. The trained second audio encoder and the audio decoder are used for speech synthesis.

2. The method according to claim 1, characterized in that, The step of updating the model parameters of the audio codec model using the reconstructed spectrum and the spectrum representation includes: The value of the loss function is determined using the reconstructed spectrum and the spectrum representation. The model parameters of the audio codec model are updated using the value of the loss function; The value of the loss function is determined by at least two of the first, second, and third loss function terms; the value of the first loss function term is obtained by the difference between the reconstructed spectrum and the spectral representation; the value of the second loss function term is obtained by the KL divergence between the distribution of the latent space feature representation and the standard normal distribution; and the value of the third loss function term is obtained by the probability that the discriminator distinguishes between the reconstructed spectrum and the spectral representation.

3. The method according to claim 1, characterized in that, The second audio encoder includes multiple fully connected layers; The step of mapping the acoustic feature representation to the latent space using the second audio encoder to obtain the latent space feature representation includes: using the multi-layer fully connected layer to predict the distribution parameters corresponding to each time frame based on the acoustic feature representation, the distribution parameters including mean and variance; performing reparameterization sampling based on the distribution parameters corresponding to each time frame, introducing Gaussian noise in the reparameterization sampling; and using the latent space variables obtained by the reparameterization sampling to obtain the latent space feature representation.

4. The method according to claim 2, characterized in that, The value of the second loss function term is obtained by adding a marginal constraint to the KL divergence between the latent space feature representation distribution and the standard normal distribution.

5. A speech synthesis method, characterized in that, The method includes: Obtain the text to be synthesized; The text feature representation of the text to be synthesized is extracted using a text encoder; The text feature representation is mapped to an acoustic feature representation using an acoustic encoder; Using a second audio encoder, the acoustic feature representation is mapped into the latent space to obtain the latent space feature representation; The latent space feature representation is decoded using an audio decoder to obtain the speech corresponding to the text to be synthesized; The second audio encoder and the audio decoder are pre-trained using the method described in any one of claims 1 to 4.

6. A training device for an audio codec model, characterized in that, The audio encoding / decoding model includes a first audio encoder, a second audio encoder, and an audio decoder; the device includes: The sample acquisition unit is configured to acquire training data including multiple audio samples; A model training unit is configured to train the audio codec model using the training data; wherein the training includes: extracting the spectral representation of the audio sample and inputting the spectral representation into the first audio encoder; extracting the acoustic feature representation of the audio sample using the first audio encoder; mapping the acoustic feature representation into the latent space using the second audio encoder to obtain the latent space feature representation of the audio sample; performing masking processing on the latent space variables corresponding to at least one time frame in the latent space feature representation to obtain the masked latent space feature representation, the masking processing including short-span masking and long-span masking, wherein the masking time span of the short-span masking is shorter than that of the long-span masking, and the short-span masking has more cases than the long-span masking; performing decoding processing using the audio decoder based on the masked latent space feature representation to obtain the reconstructed spectrum; and updating the model parameters of the audio codec model using the reconstructed spectrum and the spectral representation. The trained second audio encoder and the audio decoder are used for speech synthesis.

7. A speech synthesis device, characterized in that, The device includes: The acquisition unit is configured to acquire the text to be synthesized. A text encoding unit is configured to extract text feature representations of the text to be synthesized using a text encoder; An acoustic coding unit is configured to map the text feature representation to an acoustic feature representation using an acoustic encoder; The mapping unit is configured to use a second audio encoder to map the acoustic feature representation to the latent space to obtain the latent space feature representation; The decoding unit is configured to decode the latent space feature representation using an audio decoder to obtain the speech corresponding to the text to be synthesized; wherein the second audio encoder and the audio decoder are pre-trained using the method described in any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 5.

9. An electronic device, characterized in that, include: One or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 5.

Images