Speech compression method and system based on feature-weighted residual vector quantization

Abstract

The application discloses a speech compression method and system based on feature-weighted residual vector quantization, and belongs to the technical field of speech compression. The method comprises the following steps: encoding a speech signal to obtain an encoded feature; calculating a feature weight of the encoded feature; performing multi-stage quantization on the encoded feature, introducing the feature weight of the encoded feature in each quantization process, calculating a weighted Euclidean distance between an input vector of each stage and a candidate code word, selecting the candidate code word with the minimum weighted Euclidean distance as a quantized vector of each stage to obtain a quantized feature; wherein the input vector of the first stage quantization is the encoded feature, and the input vectors of the remaining stage quantizations are all residuals of the encoded feature and all the quantized vectors of the previous stages; and determining a synthesized speech signal according to the quantized feature. The speech synthesis quality is ensured, and the technical problem that the speech synthesis quality is limited by the current method is solved.

Description

Technical Field

[0001] This invention relates to the field of speech compression coding technology, and in particular to a speech compression method and system based on feature-weighted residual vector quantization. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Low-rate speech coding technology has a wide range of applications in key areas such as satellite communication, shortwave communication, underwater acoustic communication, and secure communication.

[0004] Current methods mainly synthesize original speech through speech coding models. The process includes: encoding the speech at the encoding end to obtain coding features, quantizing and dequantizing the coding features to obtain dequantized features, and decoding the dequantized features to obtain synthesized speech.

[0005] However, existing speech compression methods use a uniform distance metric when quantizing encoded features, without distinguishing the importance of different feature dimensions. This results in key speech features and redundant features being compressed simultaneously under low bit rate compression conditions, leading to the loss of speech detail information and failing to guarantee the quality of speech synthesis. Summary of the Invention

[0006] To address the aforementioned problems, this invention proposes a speech compression method and system based on feature-weighted residual vector quantization, which improves the quality of speech synthesis.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] Firstly, a speech compression method based on feature-weighted residual vector quantization is proposed, including:

[0009] Encode the speech signal to obtain its encoded features;

[0010] Calculate the feature weights of the encoded features;

[0011] The encoded features are quantized in multiple levels, and feature weights are introduced in each level of quantization. The weighted Euclidean distance between the input vector of each level and the candidate codeword is calculated. The candidate codeword with the smallest weighted Euclidean distance is selected as the quantization vector of each level to obtain the quantized features. The input vector of the first level of quantization is the encoded feature, and the input vectors of the remaining levels of quantization are the residuals between the encoded feature and the quantization vectors of all previous levels.

[0012] Based on the quantized features, the synthesized speech signal is determined.

[0013] Furthermore, the process of encoding the speech signal to obtain encoded features includes:

[0014] Multi-level downsampling is performed on the speech signal to obtain the downsampled features;

[0015] Multi-head attention extraction is performed on the downsampled features to obtain attention-enhanced features;

[0016] The attention-enhanced features are residually concatenated with the downsampled features to obtain the residual concatenated features;

[0017] The residual features are convolved to obtain the encoded features.

[0018] Furthermore, the process of extracting attention-enhanced features from the downsampled features through multi-head attention extraction includes:

[0019] The encoded features are divided into windows along the time dimension to obtain multiple window features;

[0020] Calculate the query features, key features, and value features for each window feature, and divide the query features, key features, and value features into multiple attention heads along the channel dimension;

[0021] Calculate the attention features for each attention head;

[0022] The attention features of each attention head are concatenated along the channel dimension to obtain attention enhancement features.

[0023] Furthermore, the process of calculating the feature weights of the encoded features includes:

[0024] Channel compression is performed on the encoded features to obtain compressed features;

[0025] By modeling the time dimension of the compressed features through depthwise separable convolution, we obtain the time-modeled features.

[0026] After time modeling, the features are mapped to a scalar value representing the attention weight, which serves as the feature weight for encoding the features.

[0027] Furthermore, the feature weights of the encoded features are aligned with the time dimension of the encoded features to obtain the aligned weights;

[0028] The aligned weights are modulated by modulation coefficients to obtain the modulation weights used in each quantization stage.

[0029] The modulation weights are used as weighting weights to calculate the weighted Euclidean distance between the input vector at each level and the candidate codeword.

[0030] Furthermore, the quantized features are dequantized to obtain the dequantized features;

[0031] The dequantized features are then decoded to obtain the synthesized speech signal.

[0032] Secondly, a speech compression system based on feature-weighted residual vector quantization is proposed, including:

[0033] The encoding module is used to encode speech signals and obtain encoded features;

[0034] The feature weight generation module is used to calculate the importance weights of different feature components in the encoded features and obtain the feature weights of the encoded features.

[0035] The quantization module is used to perform multi-level quantization on the encoded features. In each quantization process, feature weights of the encoded features are introduced, and the weighted Euclidean distance between the input vector of each level and the candidate codeword is calculated. The candidate codeword with the smallest weighted Euclidean distance is selected as the quantization vector of each level to obtain the quantized features. The input vector of the first level of quantization is the encoded feature, and the input vectors of the remaining levels of quantization are the residuals between the encoded feature and the quantization vectors of all previous levels.

[0036] The decoding module is used to determine the synthesized speech signal based on the quantized features.

[0037] Thirdly, a computer device is proposed, the device comprising:

[0038] A processor, adapted to execute computer programs;

[0039] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the speech compression method based on feature-weighted residual vector quantization proposed in the first aspect.

[0040] Fourthly, a computer-readable storage medium is proposed, wherein the computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed by the speech compression method based on feature-weighted residual vector quantization proposed in the first aspect.

[0041] Fifthly, a computer program product is proposed, which includes a computer program that, when executed by a processor, implements the speech compression method based on feature-weighted residual vector quantization proposed in the first aspect.

[0042] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0043] This invention proposes a speech compression method and system based on feature-weighted residual vector quantization. When quantizing the encoded features of a speech signal, the method incorporates the weights of the encoded features into the calculation of the weighted Euclidean distance between each level of input vector and candidate codewords. This strengthens the influence of key information features in the quantization process, thereby achieving accurate quantization of the encoded features based on the weighted Euclidean distance and ensuring the quality of the synthesized speech.

[0044] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0045] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0046] Figure 1 This is a block diagram of the overall structure of the speech coding model proposed in this embodiment of the invention;

[0047] Figure 2 This is a flowchart of the multi-level quantization process for coding features proposed in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the overall process of the speech compression method based on feature-weighted residual vector quantization proposed in an embodiment of the present invention;

[0049] Figure 4 This is a flowchart of the attention enhancement process proposed in an embodiment of the present invention;

[0050] Figure 5 This is a flowchart of the feature weight generation process proposed in an embodiment of the present invention. Detailed Implementation

[0051] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0052] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0053] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0054] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0055] It should be noted that all data acquisition is conducted in accordance with laws and regulations and with user consent, and the data is used legally.

[0056] First, the application scenarios of the speech compression method based on feature-weighted residual vector quantization proposed in the embodiments of the present invention will be described.

[0057] The speech compression method based on feature-weighted residual vector quantization proposed in this invention is applied to the speech compression application scenario.

[0058] Low-bit-rate speech coding technology has wide applications in several key fields, including satellite communication, shortwave communication, underwater acoustic communication, and secure communication, especially in bandwidth-constrained or extremely harsh environments where the demand for speech synthesis technology is increasing. For example, satellite communication often needs to cross the atmosphere and space, where signal transmission bandwidth is extremely limited. In such cases, low-bit-rate speech coding can significantly save transmission bandwidth and improve communication efficiency. In encrypted communication, low-bit-rate speech coding technology can not only compress data but also enhance speech security through encryption and encoding. In extremely harsh mountainous communication environments, communication equipment must withstand extreme low temperatures and strong winds while maintaining stable operation with limited energy supplies, often requiring extremely low encoding rates for speech. As the encoding rate decreases, the quality of synthesized speech is affected. Therefore, how to ensure the quality of synthesized speech at low bit rates has become a key issue in current speech coding and decoding technology research.

[0059] The current process of voice compression includes:

[0060] (1) Encoding end: The original speech is downsampled using a convolutional neural network so that the main feature information can be preserved while compressing, thus completing the extraction of acoustic information.

[0061] (2) Quantization end: The codebook is designed to quantize the coding features output by the encoding end. The codebook is updated by calculating the Euclidean distance between the input vector and the initial codebook. The codeword with the closest Euclidean distance to the input vector is calculated and replaced with the codebook index for transmission.

[0062] (3) Dequantization end: Dequantize the quantized features output by the quantization end according to the quantization index, and restore the acoustic features according to the codebook.

[0063] (4) Decoding end: The dequantized features output by the dequantization end are upsampled using a convolutional neural network to restore the input speech and obtain the synthesized speech signal.

[0064] Speech signals exhibit significant temporal continuity and varying feature importance. Different time points and feature channels contribute significantly differently to perceptual quality during speech reconstruction. However, existing speech compression methods typically employ a uniform distance metric when quantizing encoded features, failing to differentiate the importance of different feature dimensions. This leads to the simultaneous compression of critical and redundant features at low bit rates, resulting in the loss of detailed speech information. Furthermore, current encoders primarily model speech features based on convolutional structures. While capable of extracting local temporal features, they fall short in modeling short-term continuous structures and local temporal correlations, causing blurred or disjointed details in the decoded synthesized speech. Moreover, the lack of adaptive adjustment mechanisms for different quantization layers during multi-level quantization limits overall quantization performance and stability. Therefore, there is an urgent need to explore a speech compression method that comprehensively considers differences in feature importance and local temporal correlations to improve speech synthesis quality at low bit rates.

[0065] Based on this, in order to improve the quality of speech synthesis, the embodiments of the present invention propose a speech compression method based on feature-weighted residual vector quantization, such as... Figures 1-5 As shown, it includes:

[0066] Encode the speech signal to obtain its encoded features;

[0067] Calculate the feature weights of the encoded features;

[0068] The encoded features are quantized in multiple levels, and feature weights are introduced in each level of quantization. The weighted Euclidean distance between the input vector of each level and the candidate codeword is calculated. The candidate codeword with the smallest weighted Euclidean distance is selected as the quantization vector of each level to obtain the quantized features. The input vector of the first level of quantization is the encoded feature, and the input vectors of the remaining levels of quantization are the residuals between the encoded feature and the quantization vectors of all previous levels.

[0069] Based on the quantized features, the synthesized speech signal is determined.

[0070] The speech signal obtained in this embodiment of the invention is 8kHz sampled speech, which can be obtained from... express, Where T is the number of speech channels, and T is the total number of speech sample points. Where d is the duration of the speech. The sampling rate for the speech.

[0071] Speech signals are processed using a speech coding model to obtain synthesized speech signals, such as... Figure 1 As shown, the speech coding model includes an encoder, a quantization end, a dequantization end, and a decoder. The encoder encodes the speech signal to obtain coded features and calculates the feature weights of the coded features. The quantization end introduces the feature weights of the coded features and quantizes the coded features to obtain quantized features. The dequantization end dequantizes the quantized features to obtain dequantized features. The decoder decodes the dequantized features to obtain the synthesized speech signal.

[0072] In some embodiments, the process of encoding a speech signal to obtain encoded features includes:

[0073] Multi-level downsampling is performed on the speech signal to obtain the downsampled features;

[0074] Multi-head attention extraction is performed on the downsampled features to obtain attention-enhanced features;

[0075] The attention-enhanced features are residually concatenated with the downsampled features to obtain the residual concatenated features;

[0076] The residual features are convolved to obtain the encoded features.

[0077] The process of extracting attention-enhanced features from downsampled features through multi-head attention extraction includes:

[0078] The encoded features are divided into windows along the time dimension to obtain multiple window features;

[0079] Calculate the query features, key features, and value features for each window feature, and divide the query features, key features, and value features into multiple attention heads along the channel dimension;

[0080] Calculate the attention features for each attention head;

[0081] The attention features of each attention head are concatenated along the channel dimension to obtain attention enhancement features.

[0082] In some embodiments, the process of calculating the feature weights of encoded features includes:

[0083] Channel compression is performed on the encoded features to obtain compressed features;

[0084] By modeling the time dimension of the compressed features through depthwise separable convolution, we obtain the time-modeled features.

[0085] After time modeling, the features are mapped to a scalar value representing the attention weight, which serves as the feature weight for encoding the features.

[0086] The encoder in this embodiment of the invention includes a one-dimensional convolutional downsampling backbone, a local window multi-head attention module, and a feature weight generation module. Its core lies in introducing the local window multi-head attention module and the feature weight generation module to enhance the expressive power of speech features under low bit rate conditions. First, the acquired speech signal undergoes basic temporal feature extraction through a one-dimensional convolution layer to obtain an initial speech representation. This convolutional layer has 32 output channels and a kernel size of 7. Subsequently, the encoder employs a multi-level downsampling structure to progressively compress the temporal resolution. Each downsampling level is implemented through a one-dimensional convolution with a stride of S, and the kernel size is twice the stride. The number of channels is doubled during downsampling to improve feature representation while reducing temporal resolution. The downsampling stride set is S = (1, 4, 8, 10), and the downsampled feature x is obtained after downsampling. [B,C,T], where T is the time step length after downsampling. After multi-level downsampling, the data enters a local window multi-head attention module, which is used to model the local temporal correlation of the downsampled features and enhance the features. Attention-enhanced features are obtained by extracting attention from the downsampled features through multi-head attention. For example... Figure 4 As shown, the local window multi-head attention module performs downsampling on the feature x. The input [B,C,T] includes a window partitioning unit, a feature mapping unit, a multi-head attention computation unit, and a residual fusion unit. The window partitioning unit divides the downsampled input features into multiple local time windows of length W along the time dimension, and pads the tail features that are less than the integer window length so that the padded length P is divisible by W, thus obtaining multiple window features. The number of windows is m = P / W, and the window features can be represented as [B,C,m,W]. The feature mapping unit uses one-dimensional convolution to perform a linear transformation on the window features, generating query features, key features, and value features, denoted as Q, K, and V, respectively. [B,C,P]. Subsequently, the multi-head attention computation unit divides Q, K, and V into H attention heads along the channel dimension, with each attention head having a channel dimension of d=C / H, thus forming Q, K,V at the window granularity. [B,H,m,W,d]. Within each window, each attention head independently calculates its attention weights and performs a weighted summation on the corresponding value features V to obtain the attention features for each attention head. The attention features of each attention head are then concatenated along the channel dimension to reconstruct [B,C,P], and channel mapping is performed using a 1×1 one-dimensional convolution to obtain the attention enhancement feature f. [B,C,T] (restores the original length T after padding). The residual fusion unit performs a residual concatenation between the attention-enhanced features and the downsampled features of the original input, and introduces a scaling factor to modulate the amplitude of the attention-enhanced features to ensure the stability of network training, thus obtaining the residual concatenated features. Finally, a one-dimensional convolution layer projects the residual concatenated features onto a preset dimension to obtain the encoded features z output by the encoder, whose tensor shape is [B,C,T]. Here, W is set to 8 and H is set to 4.

[0087] The encoded features then enter the feature weight generation module. This module assigns weights to each feature component of the encoded features through adaptive learning, obtaining the feature weights for subsequent weighting and adjustment during quantization. The feature weight generation module includes a feature compression unit, a temporal modeling unit, and a weight generation unit. For example... Figure 5 As shown, firstly, the encoded features z of the input module are compressed through a 1×1 convolution layer to obtain the compressed features s. [B,D,T], where D is the number of channels after compression. Next, the compressed features are modeled in the temporal dimension using depthwise separable convolutions to capture the local dependencies of the features over time. Depthwise separable convolutions include channel-wise convolutions and pointwise convolutions, which effectively reduce computation while preserving sufficient modeling of local information. The features output by this convolution are then passed through the SiLU activation function to obtain the temporally modeled features. [B,D,T]. Based on this, the feature weight generation module continues to calculate weights. Through a 1×1 convolutional layer, the features after time modeling... Attention weights are mapped to a scalar value representation. Specifically, they are generated using a tanh activation function in the range [1]. Attention weights are defined within the range [1 + α], where α is an adjustable parameter controlling the magnitude of the weights. This process employs a gating mechanism (w = 1 + α · tanh(y)) to ensure training stability and prevent unstable updates caused by excessively large or small weights. Ultimately, the generated attention weights serve as the feature weights w for encoding the features. [B,C,T] is used in the subsequent weighted quantization process.

[0088] In this embodiment of the invention, the encoded features are quantized at multiple levels at the quantization end. In each quantization process, the feature weight of the encoded features is introduced, the weighted Euclidean distance between the input vector of each level and the candidate codeword is calculated, the candidate codeword with the smallest weighted Euclidean distance is selected as the quantization vector of each level, and the quantization vectors of all levels are used as the index to obtain the quantized features.

[0089] The process of introducing feature weights for encoded features and calculating the weighted Euclidean distance between each level of input vector and candidate codeword includes:

[0090] Align the feature weights of the encoded features with the time dimension of the encoded features to obtain the aligned weights;

[0091] The aligned weights are modulated by modulation coefficients to obtain the modulation weights used in each quantization stage.

[0092] The modulation weights are used as weighting weights to calculate the weighted Euclidean distance between the input vector at each level and the candidate codeword.

[0093] This invention employs a multi-level vector quantization system at the quantization stage, comprising M=256 candidate codewords. Each candidate codeword has a vector dimension of C=512, and the number of potential frame vectors is N=25. Each codebook can encode... Bits. To ensure quantization performance and convergence speed, the K-Means clustering algorithm is used to partition the encoded latent vector space, obtaining an initialized codebook set. The input to the quantization end includes the encoded features obtained from the encoder and the feature weights w of the encoded features. [B,C,T]; To enable feature weights to participate in codebook search, the feature weights w of the encoded features need to be aligned in the time dimension to the length N of the encoded features in the time dimension, resulting in aligned weights. [B,C,N], and for Amplitude limiting and normalization are performed to ensure stability. A multi-stage quantizer is used at the quantization stage. Each stage quantizes the quantization residual of the previous stage to progressively approximate the original vector, thereby obtaining the best-matching candidate codeword. Unlike traditional residual vector quantization which uses standard Euclidean distance, this invention introduces aligned weights during the codebook search process at each quantizer stage. A weighted distance function is constructed to enhance the influence of key information features in the quantization stage, quickly and accurately locating the best-matching codebook vector, thereby minimizing quantization error. Specifically, let the first-level input vector... (i=1) is the encoded feature z, which is input into the first-level quantizer. Let the input vector of the i-th (i≥2) level be... The candidate codeword is the residual between the encoded feature z and the quantization vectors of all levels up to level i (i≥2). The following weighted Euclidean distance calculation formula is used to calculate the weighted Euclidean distance between each level of input vector and candidate codeword:

[0094] ;

[0095] in, For input vector with candidate codewords The weighted Euclidean distance between them For the input vector, Candidate codewords, The weights are used for quantization. In each quantization level, the candidate codeword with the smallest weighted Euclidean distance is output as the quantization vector for that level.

[0096] The process of a multi-level quantizer performing multi-level quantization on encoded features includes:

[0097] First, the first-level quantizer quantizes the encoded feature z to obtain the first-level quantization vector. The second-level quantizer stores the residual between the encoded features and the quantized vector of the first level. Quantization is performed to obtain the second-level quantization vector. The residual between the encoded features and the quantized vectors of the first and second levels by the third-level quantizer. Quantization is performed to obtain the third-level quantization vector. The remaining quantizers iterate sequentially until all quantizations are complete.

[0098] To further conform to the quantization principle of residual approximation at each stage, this embodiment of the invention introduces a decreasing weight modulation strategy in the multi-stage residual vector quantization process. The weighting weights used in different quantization stages are controlled hierarchically. The aligned weights are modulated using modulation coefficients to obtain the modulation weights used in each quantization stage. These modulation weights are then used as weighting weights to calculate the weighted Euclidean distance between the input vector and the candidate codeword at each stage. Specifically:

[0099] In the i-th level quantization, a modulation coefficient is introduced. For the aligned weights Perform scaling modulation to obtain modulation weights :

[0100] ;

[0101] And limit the modulation coefficient It decreases progressively with increasing quantization level. Among them, The values ​​are pre-set to a decreasing sequence (from 1.0 to 0.2) and are adaptively optimized as learnable parameters during training.

[0102] This approach strengthens the influence of feature weights on codebook search in the lower-level quantization stages, while gradually weakening the weight modulation intensity in the higher-level residual quantization stages. This makes the quantization process approximate the standard Euclidean distance metric, thereby improving the overall stability and robustness of the quantization process. After quantization, the quantization vectors of all levels are used as indices to obtain the quantized features. It is compressed and transmitted to the decoding end.

[0103] In some embodiments, the quantized features are dequantized to obtain dequantized features;

[0104] The dequantized features are then decoded to obtain the synthesized speech signal.

[0105] In this embodiment of the invention, the quantized features are packaged into a binary byte stream and transmitted to the dequantization end.

[0106] The dequantization end obtains the dequantized features in a multi-level quantizer based on the index, and then adds them sequentially to obtain the final latent frame vector, i.e., the dequantized features. , .

[0107] Dequantized features The input is fed into the decoder to reconstruct the speech, obtaining a synthesized speech signal. The process of the decoder reconstructing the speech includes: the decoder and encoder have symmetrical structures; however, the decoder differs from the encoder in that it lacks a local window multi-head attention module and a feature weight generation module. The process of the decoder reconstructing the speech to obtain a synthesized speech signal includes:

[0108] First, the dequantized features are processed by a one-dimensional convolution with a kernel size of 7 and 512 channels. The obtained feature chain is then continuously upsampled to gradually restore the temporal resolution and obtain the upsampled features. The upsampling uses a one-dimensional transposed convolution with a stride of E and a kernel size twice the stride of E. During the upsampling process, the number of channels is halved, where E = (10, 8, 4, 1). The upsampled features are then processed by a one-dimensional convolution with a kernel size of 7 and 32 channels to obtain the synthesized speech signal.

[0109] To further improve the speech compression quality of the speech coding model, a multi-scale STFT discriminator (MS-STFT) and a multi-period discriminator (MPD) are used to discriminate the quality of the synthesized speech signal during the training phase of the speech coding model. These discriminators are used only during training. Specifically, the discriminator's main function is to judge the performance of the speech coding model, training it to generate better synthesized speech signals. The discriminator's input is the input speech signal from the encoder and the output synthesized speech signal from the decoder. The input speech signal is the original speech. By predicting the authenticity of the synthesized speech signal, it learns how to distinguish between the original and synthesized speech signals and feeds back the prediction result of the authenticity of the synthesized speech signal to the speech coding model, allowing them to compete and cooperate in an adversarial training manner. The MS-STFT discriminator consists of multiple sub-discriminators with the same structure, each operating on a complex STFT representation at a different scale. Specifically, it employs five sub-discriminators, corresponding to five different Short-Time Fourier Transform (STFT) window lengths: 2048, 1024, 512, 256, and 128, with sliding step sizes of 512, 256, 128, 64, and 32, respectively. Each sub-discriminator operates at a different time scale and frequency bandwidth to fully extract local details and global structural information of the speech signal. After the input speech signal undergoes STFT transformation, the real and imaginary parts of its complex spectrum are extracted and concatenated along the channel dimension to form a two-dimensional input tensor with the shape [B, 2, F, T], where F is the frequency dimension and T is the time frame number. This complex spectrum is then input into a structurally consistent two-dimensional convolutional network for discrimination. The two-dimensional convolutional network structure includes: a two-dimensional convolutional layer with a kernel size of 3×8 and 32 channels; several two-dimensional convolutional layers with different temporal dilation rates D and frequency strides of 2; and a final two-dimensional convolutional layer (kernel size 3×3, stride 1×1) used to determine whether a data type is true or false. The discriminator at each scale outputs a set of true / false classifications and intermediate feature maps, and the results of all sub-discriminators are combined for training.

[0110] MPD is a hybrid architecture consisting of multiple sub-discriminators, each focusing on modeling speech patterns under a specific period *p*, thereby improving the ability to perceive different periodic features. Specifically, the sub-discriminators capture the different implicit structures of each other by examining different parts of the input audio. The period *p* is set to [2, 3, 5, 7, 11] to avoid overlap and structural redundancy. First, the one-dimensional original speech of length *T* is reconstructed into a two-dimensional matrix of height *T / p* and width *p*. Then, two-dimensional convolutions are applied to the reconstructed two-dimensional matrix, followed by a LeakyReLU activation function after each convolutional layer. All convolutional layers operate in the time dimension, and the kernel size on the width axis is limited to 1 to process periodic samples independently. Each sub-discriminator outputs a feature map of a real and a generated sample after each convolutional layer, and the outputs of all sub-discriminators are used uniformly for training.

[0111] In summary, MS-STFT and MPD form a complementary discrimination mechanism in the frequency and time domains, providing the decoder with more stable and multi-faceted training signals, which helps to improve the naturalness and clarity of the synthesized speech signal.

[0112] The speech compression method based on feature-weighted residual vector quantization proposed in this invention jointly models the encoded features in both the temporal and channel dimensions, adaptively learning the importance weights of different feature components in the encoded features as feature weights. During the quantization process, these feature weights are introduced to construct a weighted Euclidean distance function, giving key information features in the encoded features higher influence weights during the quantization stage. Simultaneously, a multi-level residual quantization structure and a decreasing weight modulation strategy are combined to improve quantization accuracy and stability. The local window multi-head attention module divides the encoded features into multiple local windows along the temporal dimension and performs multi-head attention calculations within each window to capture temporally relevant features within a short time range. Simultaneously, residual connections are used to fuse attention-enhanced features and downsampled features, thereby enhancing the model's ability to model short-term continuous structures and preserve speech details. This improves the ability to preserve speech details under low bitrate conditions and ensures the quality of speech synthesis under extremely low bitrate conditions.

[0113] This invention proposes a speech compression system based on feature-weighted residual vector quantization, comprising:

[0114] The encoding module is used to encode speech signals and obtain encoded features;

[0115] The feature weight generation module is used to calculate the importance weights of different feature components in the encoded features and obtain the feature weights of the encoded features.

[0116] The quantization module is used to perform multi-level quantization on the encoded features. In each quantization process, feature weights of the encoded features are introduced, and the weighted Euclidean distance between the input vector of each level and the candidate codeword is calculated. The candidate codeword with the smallest weighted Euclidean distance is selected as the quantization vector of each level to obtain the quantized features. The input vector of the first level of quantization is the encoded feature, and the input vectors of the remaining levels of quantization are the residuals between the encoded feature and the quantization vectors of all previous levels.

[0117] The decoding module is used to determine the synthesized speech signal based on the quantized features.

[0118] It should be noted that the speech compression system based on feature-weighted residual vector quantization provided in the above embodiments is only illustrated by the division of the above functional modules during speech encoding. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the speech compression system based on feature-weighted residual vector quantization provided in the above embodiments and the speech compression method embodiments based on feature-weighted residual vector quantization belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0119] The present invention also discloses a computer device, the device comprising:

[0120] A processor, adapted to execute computer programs;

[0121] A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the speech compression method based on feature-weighted residual vector quantization disclosed in the embodiments of the present invention.

[0122] The computer device can be a portable mobile terminal, such as a smartphone, tablet, laptop, or desktop computer. Typically, a computer device includes a processor and memory.

[0123] The present invention also discloses a computer-readable storage medium storing a computer program adapted for loading and execution by a processor of the speech compression method based on feature-weighted residual vector quantization disclosed in the embodiments of the present invention.

[0124] The present invention also discloses a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the speech compression method based on feature-weighted residual vector quantization disclosed in the embodiments of the present invention.

[0125] The method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0126] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0127] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A speech compression method based on feature-weighted residual vector quantization, characterized in that, include: Encode the speech signal to obtain its encoded features; Channel compression is performed on the encoded features to obtain compressed features; By modeling the time dimension of the compressed features using depthwise separable convolution, we obtain the time-modeled features. [B,D,T]; After temporal modeling, the features are sequentially passed through a 1×1 convolutional layer and a tanh activation function to generate a range within... The attention weights between the elements are calculated, where α is an adjustable parameter used to control the magnitude of the weights, resulting in the feature weights w for encoding the features. [B,C,T]; Where D is the number of channels after compression, and T is the time step length after downsampling; The encoded features are quantized in multiple levels, and feature weights are introduced in each level of quantization. The weighted Euclidean distance between the input vector of each level and the candidate codeword is calculated. The candidate codeword with the smallest weighted Euclidean distance is selected as the quantization vector of each level to obtain the quantized features. The input vector of the first level of quantization is the encoded feature, and the input vectors of the remaining levels of quantization are the residuals between the encoded feature and the quantization vectors of all previous levels. A decreasing weight modulation strategy is introduced in the multi-level residual vector quantization process to hierarchically control the weights used in different quantization levels. The feature weights of the encoded features are aligned to the length of the encoded features in the time dimension to obtain the aligned weights. The aligned weights are then modulated by modulation coefficients to obtain the modulation weights used in each quantization level. The modulation weights are used as weighting weights to calculate the weighted Euclidean distance between the input vector and the candidate codeword at each level. Specifically: In the i-th level quantization, a modulation coefficient is introduced. For the aligned weights Perform scaling modulation to obtain modulation weights : ； And limit the modulation coefficient The value decreases progressively with increasing quantization level; among which, The sequence is pre-set as a decreasing sequence and is adaptively optimized during training as a learnable parameter. Based on the quantized features, the synthesized speech signal is determined.

2. The speech compression method based on feature-weighted residual vector quantization as described in claim 1, characterized in that, The process of encoding speech signals to obtain encoded features includes: Multi-level downsampling is performed on the speech signal to obtain the downsampled features; Multi-head attention extraction is performed on the downsampled features to obtain attention-enhanced features; The attention-enhanced features are residually concatenated with the downsampled features to obtain the residual concatenated features; The residual features are convolved to obtain the encoded features.

3. The speech compression method based on feature-weighted residual vector quantization as described in claim 2, characterized in that, The process of extracting attention-enhanced features from downsampled features using multi-head attention includes: The encoded features are divided into windows along the time dimension to obtain multiple window features; Calculate the query features, key features, and value features for each window feature, and divide the query features, key features, and value features into multiple attention heads along the channel dimension; Calculate the attention features for each attention head; The attention features of each attention head are concatenated along the channel dimension to obtain attention enhancement features.

4. The speech compression method based on feature-weighted residual vector quantization as described in claim 1, characterized in that, The quantized features are dequantized to obtain the dequantized features. The dequantized features are then decoded to obtain the synthesized speech signal.

5. A speech compression system based on feature-weighted residual vector quantization, characterized in that, include: The encoding module is used to encode speech signals and obtain encoded features; The feature weight generation module is used to calculate the importance weights of different feature components in the encoded features and obtain the feature weights of the encoded features. Channel compression is performed on the encoded features to obtain compressed features; By modeling the time dimension of the compressed features using depthwise separable convolution, we obtain the time-modeled features. [B,D,T]; After temporal modeling, the features are sequentially passed through a 1×1 convolutional layer and a tanh activation function to generate a range within... The attention weights between the elements are calculated, where α is an adjustable parameter used to control the magnitude of the weights, resulting in the feature weights w for encoding the features. [B,C,T]; where D is the number of channels after compression, and T is the time step length after downsampling; The quantization module performs multi-level quantization on the encoded features. In each quantization level, feature weights are introduced to calculate the weighted Euclidean distance between the input vector and candidate codewords. The candidate codeword with the smallest weighted Euclidean distance is selected as the quantization vector for each level, yielding the quantized features. The input vector for the first quantization level is the encoded feature, while the input vectors for subsequent levels are the residuals between the encoded feature and all previous quantization vectors. A decreasing weight modulation strategy is introduced during the multi-level residual vector quantization process. The weights used in different quantization levels are hierarchically controlled, aligning the feature weights of the encoded features to their time length, resulting in aligned weights. These aligned weights are then modulated using modulation coefficients to obtain the modulation weights used in each quantization level. These modulation weights are then used as weights to calculate the weighted Euclidean distance between the input vector and candidate codewords at each level. Specifically: In the i-th level quantization, a modulation coefficient is introduced. For the aligned weights Perform scaling modulation to obtain modulation weights : ； And limit the modulation coefficient The value decreases progressively with increasing quantization level; among which, The sequence is pre-set as a decreasing sequence and is adaptively optimized during training as a learnable parameter. The decoding module is used to determine the synthesized speech signal based on the quantized features.

6. An electronic device, characterized in that, The device includes: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the speech compression method based on feature-weighted residual vector quantization as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and executed by the speech compression method based on feature-weighted residual vector quantization as described in any one of claims 1-4.

8. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the speech compression method based on feature-weighted residual vector quantization as described in any one of claims 1-4.

Images