Speech attribute decomposition quantification system and method with high compression ratio

Abstract

The invention discloses a voice attribute decomposition quantization system and method with a high compression ratio, and relates to the technical field of voice quantization, and the system comprises a generator composed of a feature extractor, an encoder, a decoupler and a decoder, and a discriminator. The feature extractor is used for extracting two-dimensional hidden features according to the audio signal to be quantized; the encoder is used for encoding the two-dimensional hidden features to generate hidden features; the decoupler is used for carrying out feature extraction on the hidden features to obtain a plurality of discrete tokens; the decoder is used for obtaining a reconstructed audio according to the plurality of discrete tokens; and the discriminator is used for assisting the generator in parameter updating. According to the method, through fine selection of the input features, enhancement of the expressive power of a decoder and introduction of various discriminators, generation of a voice quantization token with a high compression rate can be effectively guaranteed, on one hand, the data storage space and the LLM calculation amount are reduced, and on the other hand, modeling of a large model for an ultra-long audio can be promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of speech quantization, and more particularly to a speech attribute decomposition and quantization system and method with a high compression rate. Background Art

[0002] Traditionally, speech quantization is a technology that converts speech signals into discrete digital forms for compression for storage and transmission, and is widely used in communication systems. With the development of deep learning, speech quantization technology based on deep neural networks has also been gradually applied to speech recognition, speech synthesis, and other occasions that require processing speech signals. By converting continuous speech signals into discrete tokens, large language models (LLMs) can be applied to the field of speech generation and significantly improve the results of speech generation-related tasks, especially zero-sample speech synthesis tasks, that is, given a piece of reference audio and text, the model can generate speech with the timbre, style, and corresponding content of the reference audio.

[0003] However, there are still two obvious problems with current speech quantization, which limit the further development of LLM in the field of speech. First, the compression rate is not high. For a one-second speech signal, the current quantization model generally generates hundreds of tokens. Too many tokens will increase the difficulty of LLM modeling, and will also increase the amount of calculation, further limiting the ability to model long audio. On the other hand, speech contains multiple attribute components such as rhythmic style, timbre, and content. The current quantization model compresses this information uniformly, which makes it difficult to model speech timbre or style transfer. A typical example is that the speech timbre generated during zero-shot TTS is not completely similar to the reference audio. Therefore, how to meet the requirements for timbre replication when building the LLM model under high compression rate is an urgent problem that technicians in this field need to solve. Summary of the invention

[0004] In view of this, the present invention provides a speech attribute decomposition and quantization system and method with high compression rate, which overcomes the above-mentioned defects.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A high compression rate speech attribute decomposition and quantization system, comprising a generator and a discriminator consisting of a feature extractor, an encoder, a decoupler and a decoder;

[0007] The feature extractor is used to extract two-dimensional latent features according to the audio signal to be quantized;

[0008] The encoder is used to encode the two-dimensional latent feature to generate a latent feature;

[0009] The decoupler is used to extract features from the latent features to obtain multiple discrete tokens;

[0010] The decoder is used to obtain reconstructed audio according to the plurality of discrete tokens;

[0011] The discriminator is used to assist the generator in updating parameters.

[0012] Optionally, the feature extractor includes a pre-convolution layer and a feature fusion layer, the pre-convolution layer is used to down-sample the audio signal to be quantized to obtain preliminary features; the feature fusion layer is used to process the preliminary features to generate two-dimensional latent features.

[0013] Optionally, the encoder includes a multi-layer one-dimensional convolutional network and a Transformer model; the one-dimensional convolutional network is used to downsample the two-dimensional latent features in the time dimension to obtain compressed features; the Transformer model is used to capture the long-distance dependencies of the compressed features using a self-attention mechanism and process them to generate latent features.

[0014] Optionally, the decoupler includes a timbre extractor, a content decoupler and a rhythm decoupler; the timbre extractor is used to convert the latent features into timbre features; the content decoupler is used to compress and filter the latent features, and quantize the compressed and filtered latent features into content tokens; the rhythm decoupler is used to compress and filter the latent features, and quantize the compressed and filtered latent features into rhythm tokens.

[0015] Optionally, the content decoupler includes a compression module, a filtering module and a conversion module, the compression module is used to downsample the latent features using multi-layer convolution to obtain preliminary compressed features; the filtering module is used to perform text prediction and rhythm and timbre filtering on the preliminary compressed features to obtain content latent features; the conversion module is used to quantify the content latent features into content tokens.

[0016] Optionally, the decoder includes an upsampling module and a diffusion model, the upsampling module is used to restore the time length of the input feature to the time length of the audio signal to be quantized to generate an upsampled feature; the diffusion model is used to generate reconstructed audio based on the upsampled feature.

[0017] Optionally, the discriminator includes a multi-scale discriminator, a dynamic granularity discriminator, and a multi-period discriminator; the multi-scale discriminator is used to perform semantic discrimination at different levels on the reconstructed audio or the audio signal to be quantized; the dynamic granularity discriminator is used to perform speech discrimination at different granularities on the reconstructed audio or the audio signal to be quantized; the multi-period discriminator is used to perform speech discrimination at different periods on the reconstructed audio or the audio signal to be quantized.

[0018] A high compression rate speech attribute decomposition and quantization method applied to the system comprises the following specific steps:

[0019] Extracting the two-dimensional latent features according to the audio signal to be quantized;

[0020] Encoding the two-dimensional latent feature to generate the latent feature;

[0021] Feature extraction is performed on the latent features to obtain multiple discrete tokens.

[0022] Through the above technical solutions, it can be known that compared with the prior art, the present invention discloses a high-compression speech attribute decomposition and quantization system and method, which can effectively ensure the generation of high-compression speech quantization tokens through the refined selection of input features, enhanced decoder expressiveness and the introduction of multiple discriminators. On the one hand, it reduces data storage space and LLM calculation amount, and on the other hand, it can promote large models to model ultra-long audio. The present invention simultaneously decomposes the characteristics of each attribute of the speech, and quantizes them separately based on the decomposed features. LLM can finely model each attribute of the speech, which can effectively solve the current LLM's unsatisfactory problems in style transfer and timbre replication. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0024] Figure 1 A schematic diagram of the structure of the speech attribute decomposition and quantization system with high compression rate provided by the present invention;

[0025] Figure 2 A schematic diagram of the structure of the content decoupler provided by the present invention;

[0026] Figure 3 A schematic diagram of the structure of the rhythm decoupler provided by the present invention;

[0027] Figure 4 A schematic diagram of the structure of a decoder provided by the present invention;

[0028] Figure 5 A schematic diagram of the structure of the discriminator provided by the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] On the one hand, this embodiment discloses a high compression rate speech attribute decomposition and quantization system. Figure 1 As shown, it includes a generator and a discriminator consisting of a feature extractor, an encoder, a decoupler and a decoder;

[0031] A feature extractor, used for extracting two-dimensional latent features according to the audio signal to be quantized;

[0032] An encoder, used to encode the two-dimensional latent features to generate latent features;

[0033] Decoupler, used to extract latent features and obtain multiple discrete tokens;

[0034] A decoder is used to reconstruct audio based on multiple discrete tokens;

[0035] The discriminator is used to assist the generator in updating parameters.

[0036] Furthermore, the quantization system of speech attribute decomposition with high compression rate includes a feature extractor, an encoder, attribute decouplers, a decoder and a discriminator. The audio to be quantized is input into the pre-trained feature extractor to extract compact speech features, namely two-dimensional latent features. The encoder further encodes the extracted features to generate latent features. The content decoupler, rhythm decoupler and timbre extractor respectively extract independent attribute features or discrete tokens from the latent features. The discrete tokens can be directly used for LLM modeling. During training, the attribute features are input into the decoder together to reconstruct the audio. The discriminator assists in the high-quality reconstruction of the audio by predicting whether the reconstructed audio is real or generated.

[0037] In one embodiment, the feature extractor includes a pre-convolution layer and a feature fusion layer; the pre-convolution layer is used to downsample the audio signal to be quantized to obtain preliminary features; the feature fusion layer is used to process the preliminary features to generate two-dimensional latent features.

[0038] Furthermore, the feature extractor includes a pre-convolution layer and a feature fusion layer composed of several transformer blocks. The input audio signal to be quantized is a one-dimensional speech signal. The pre-convolution layer is used to perform a downsampler to obtain preliminary features. The preliminary features are further processed by the feature fusion layer to obtain two-dimensional latent features with high-level semantics. The feature extractor is trained in a self-supervised manner, that is, the input speech signal is masked (randomly masking a part of the audio signal), and the model attempts to predict the content of these masked parts. In order to increase the robustness and generalization ability of the model, this embodiment uses data from various scenes such as daily conversations and speeches for training.

[0039] In one embodiment, the encoder includes a multi-layer one-dimensional convolutional network and a Transformer model; the one-dimensional convolutional network is used to downsample the two-dimensional latent features in the time dimension to obtain compressed features; the Transformer model is used to capture the long-distance dependencies of the compressed features using a self-attention mechanism and process them to generate latent features.

[0040] Furthermore, the encoder is composed of a multi-layer one-dimensional convolutional network and a Transformer model. The convolutional layer downsamples the two-dimensional latent features extracted by the feature extractor in the time dimension to further compress the speech and perform preliminary feature processing to obtain compressed features. The Transformer model uses the self-attention mechanism to capture long-distance dependencies in audio features for further feature processing to generate latent features.

[0041] In one embodiment, the decoupler includes a timbre extractor, a content decoupler and a rhythm decoupler; the timbre extractor is used to convert latent features into one-dimensional timbre features; the content decoupler is used to compress and filter the latent features, and quantize the compressed and filtered latent features into content tokens; the rhythm decoupler is used to compress and filter the latent features, and quantize the compressed and filtered latent features into rhythm tokens.

[0042] Furthermore, the timbre features are obtained through a timbre extractor based on Transformer, that is, the latent features generated by the encoder are converted into a one-dimensional feature vector through the timbre extractor, which represents the global speaker information of the speech. Due to the low dimension, this feature basically does not contain other information except timbre. During training, contrast loss is used to optimize the timbre extractor, that is, to maximize the similarity of speech samples from the same speaker and minimize the similarity of speech samples from different speakers.

[0043] In one embodiment, the content decoupler includes a compression module, a filtering module and a conversion module. The compression module is used to downsample the latent features using multi-layer convolution to obtain preliminary compressed features; the filtering module is used to perform text prediction and rhythm and timbre filtering on the preliminary compressed features to obtain content latent features; the conversion module is used to quantify the content latent features into discrete content tokens.

[0044] Furthermore, the content decoupler aims to extract content information from the latent features generated by the encoder, while other information such as rhythm and timbre are filtered out. Figure 2 As shown, this embodiment first uses multi-layer convolution to downsample the latent features to further compress and filter the information, obtain preliminary compressed features, and then make the information tend to retain the speech content attributes through the following steps: First, supervise the training of the preliminary compressed features, that is, use the preliminary compressed features to predict the text corresponding to the speech, and then calculate the text loss based on the CTC loss commonly used in speech recognition, so as to ensure that the features completely retain the content information. Secondly, in order to more ideally ensure that the generated content features do not contain timbre and rhythm information, the rhythm and timbre are filtered using the gradient back propagation method. Specifically: when filtering the timbre information, the generated features are used to classify the speaker and calculate the loss, and the negative gradient is used for updating during the gradient back propagation; when filtering the rhythm information, the content features are used to predict the pitch of the audio (the pitch is consistent with the rhythm to some extent), and the convolution layer parameters are also updated using the negative gradient during the gradient back propagation. After obtaining the independent content latent features, RVQ (Residual Vector Quantization) is used to convert the continuous latent features into discrete content tokens, which can be used for LLM modeling.

[0045] Among them, the prosody decoupler aims to extract pure prosody information from the latent features generated by the encoder, while other information such as content and timbre are filtered out. Figure 3As shown in the figure, like the content decoupler, firstly, multi-layer convolution is used to downsample the latent features generated by the encoder, so that the information tends to retain the rhythmic components of the speech; secondly, the downsampled features are used to predict the pitch corresponding to the speech, and the L1 loss is calculated with the real pitch of the speech for supervised training optimization, so as to ensure that the features completely retain the rhythmic information. In order to ensure that the generated content features do not contain timbre and content information, the content and timbre are filtered by gradient backpropagation. When filtering the timbre information, the feature is used for speaker classification and loss calculation, and negative gradients are used for updating during gradient backpropagation; when filtering the content information, the downsampled features are used to predict the text corresponding to the audio and calculate the CTC loss with the real text, and negative gradients are used to update the convolution layer parameters during gradient backpropagation. Finally, RVQ is used to convert the downsampled features into discrete rhythmic tokens.

[0046] In one embodiment, the decoder includes an upsampling module and a diffusion model. The upsampling module is used to restore the time length of the input feature to the time length of the audio signal to be quantized to generate an upsampled feature; the diffusion model is used to generate reconstructed audio based on the upsampled feature.

[0047] Furthermore, if Figure 4 As shown in the figure, the content token, rhythm token and timbre vector are concatenated together in the time dimension and input into the upsample module of the decoder to restore it to the same time length as the original audio and generate upsampled features; the upsample module is implemented by multiple convolutional layers; Diffusion uses the classic u-net structure to convert the upsampled features into reconstructed audio of the same length as the original speech audio.

[0048] In one embodiment, the discriminator includes a multi-scale discriminator, a dynamic granularity discriminator, and a multi-period discriminator; the multi-scale discriminator is used to perform semantic discrimination at different levels on the reconstructed audio or the audio signal to be quantized; the dynamic granularity discriminator is used to perform speech discrimination at different granularities on the reconstructed audio or the audio signal to be quantized; the multi-period discriminator is used to perform speech discrimination at different periods on the reconstructed audio or the audio signal to be quantized.

[0049] Furthermore, in order to improve the high-quality reconstruction of the decoder audio and further achieve low bit rate quantization, this embodiment adopts an adversarial training strategy, with the entire encoder and decoder as the generator, and then adding multiple discriminators to assist the generator in parameter update. Figure 5 As shown, this embodiment adopts three discriminators, namely a multi-scale discriminator, a dynamic granularity discriminator, and a multi-cycle discriminator.

[0050] The multi-scale discriminator consists of several sub-discriminators, each of which consists of multiple layers of CNN and one layer of full connection, responsible for semantic discrimination of different levels of speech. First, the input audio (i.e. original audio or reconstructed audio) is downsampled to the corresponding scale to retain the overall semantics of different levels, and then the corresponding sub-discriminator in the multi-scale discriminator is allowed to make judgments.

[0051] The dynamic granularity discriminator is composed of a network with higher complexity (compared with the above sub-discriminator, it contains more CNN layers and larger feature dimensions). Sub-audio fragments of any length are cut from any position in the input audio or reconstructed audio and input into the discriminator to determine whether they are real audio fragments or generated audio fragments. Therefore, the dynamic granularity discriminator can measure the effect of speech reconstruction from different granularities and ensure the local details of speech reconstruction.

[0052] The multi-cycle discriminator is composed of several sub-discriminators, each of which is also composed of multiple layers of CNN and one layer of full connection. The speech signal is a typical periodic signal. The multi-cycle discriminator measures the quality of speech reconstruction from the perspective of period. Specifically, the one-dimensional speech signal is first converted into two-dimensional data according to a certain step size (i.e., the number of periods), and then input into the corresponding period discriminator for judgment.

[0053] On the other hand, this embodiment also discloses a high compression rate speech attribute decomposition and quantization method, the specific steps are:

[0054] Step 1: extracting two-dimensional latent features according to the audio signal to be quantized;

[0055] Step 2: Encode the two-dimensional latent features to generate latent features;

[0056] Step 3: Extract the latent features and obtain multiple discrete tokens.

[0057] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0058] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A high compression rate speech attribute decomposition and quantization system, characterized in that: It includes a generator and a discriminator consisting of a feature extractor, an encoder, a decoupler and a decoder; The feature extractor is used to extract two-dimensional latent features according to the audio signal to be quantized; The encoder is used to encode the two-dimensional latent feature to generate a latent feature; The decoupler is used to extract features from the latent features to obtain multiple discrete tokens; The decoder is used to obtain reconstructed audio according to the plurality of discrete tokens; The discriminator is used to assist the generator in updating parameters.

2. A high compression rate speech attribute decomposition and quantization system according to claim 1, characterized in that: The feature extractor comprises a pre-convolution layer and a feature fusion layer, wherein the pre-convolution layer is used to downsample the audio signal to be quantized to obtain preliminary features; The feature fusion layer is used to process the preliminary features to generate two-dimensional latent features.

3. A high compression rate speech attribute decomposition and quantization system according to claim 1, characterized in that: The encoder includes a multi-layer one-dimensional convolutional network and a Transformer model; the one-dimensional convolutional network is used to downsample the two-dimensional latent features in the time dimension to obtain compressed features; The Transformer model is used to capture and process the long-distance dependencies of the compressed features using a self-attention mechanism to generate latent features.

4. A high compression rate speech attribute decomposition and quantization system according to claim 1, characterized in that: The decoupler includes a timbre extractor, a content decoupler and a rhythm decoupler; the timbre extractor is used to convert the latent features into timbre features; the content decoupler is used to compress and filter the latent features, and quantize the compressed and filtered latent features into content tokens; the rhythm decoupler is used to compress and filter the latent features, and quantize the compressed and filtered latent features into rhythm tokens.

5. A high compression rate speech attribute decomposition and quantization system according to claim 4, characterized in that: The content decoupler includes a compression module, a filtering module and a conversion module. The compression module is used to downsample the latent features using multi-layer convolution to obtain preliminary compressed features; the filtering module is used to perform text prediction and rhythm and timbre filtering on the preliminary compressed features to obtain content latent features; The conversion module is used to quantify the content latent features into content tokens.

6. A high compression rate speech attribute decomposition and quantization system according to claim 1, characterized in that: The decoder includes an upsampling module and a diffusion model. The upsampling module is used to restore the time length of the input feature to the time length of the audio signal to be quantized to generate an upsampled feature; the diffusion model is used to generate reconstructed audio according to the upsampled feature.

7. A high compression rate speech attribute decomposition and quantization system according to claim 1, characterized in that: The discriminator includes a multi-scale discriminator, a dynamic granularity discriminator, and a multi-period discriminator; the multi-scale discriminator is used to perform semantic discrimination at different levels on the reconstructed audio or the audio signal to be quantized; the dynamic granularity discriminator is used to perform speech discrimination at different granularities on the reconstructed audio or the audio signal to be quantized; the multi-period discriminator is used to perform speech discrimination at different periods on the reconstructed audio or the audio signal to be quantized.

8. A high compression rate speech attribute decomposition and quantization method applied in the system according to any one of claims 1 to 7, characterized in that: The specific steps are: Extracting the two-dimensional latent features according to the audio signal to be quantized; Encoding the two-dimensional latent feature to generate the latent feature; Feature extraction is performed on the latent features to obtain multiple discrete tokens.

Images