A neural network-based lightweight streaming speech coding system and method

Abstract

The application provides a neural network-based lightweight streaming speech coding system and method, which comprises an encoding compression end and a decoding reconstruction end; the encoding compression end comprises a speech encoder and a quantizer; the speech encoder comprises a time-frequency conversion module, a feature extraction module, an encoding end channel conversion module and a long-range time domain correlation extraction module; the feature extraction module comprises a time-frequency feature extraction submodule and a time-frequency size downsampling submodule; the quantizer comprises two or more layers of quantization modules, each layer of quantization module comprising an encoding end first domain conversion module, an encoding end vector quantization module, an encoding end inverse vector quantization module and an encoding end second domain conversion module; and the decoding reconstruction end comprises an inverse quantizer and a speech decoder. The application can effectively extract multi-scale feature information in a speech signal, so that high-quality speech compression coding and reconstruction can be realized at an extremely low code rate, and the parameter quantity is small, the complexity is low, and streaming coding and decoding can be realized.

Description

Technical Field

[0001] This invention relates to the field of speech coding, and in particular to a lightweight streaming speech coding system and method based on neural networks. Background Technology

[0002] For a long time, speech coding has been the cornerstone of digital signal processing, enabling efficient transmission and storage of speech signals in real-time communication systems. Traditional speech codecs perform well at medium bit rates, but their performance drops sharply at low and ultra-low bit rates, which severely limits applications such as web conferencing, cloud gaming, and interactive communication.

[0003] In recent years, advancements in neural speech coding technology have yielded significant results in addressing these challenges. By jointly learning compact latent representations through an encoder-quantizer-decoder architecture, neural speech codecs have substantially improved the perceptual quality of encoded speech, particularly in low bitrate scenarios. Among various quantization techniques, residual vector quantization (RVQ) has become the mainstream technique for discretizing latent spaces because it can effectively capture multi-stage structured residual information.

[0004] However, two key challenges remain. First, many encoder-decoder backbones still lack the ability to adequately model multi-scale and contextual speech structures, limiting their ability to extract compact representations and thus constraining rate-distortion performance. Second, standard RVQ processes often face difficulties after the first stage: residuals become irregular and misaligned with codebook vectors, leading to increased quantization loss, inefficient codebook utilization, and potential representation redundancy. Although recent research has proposed techniques to improve codebook utilization and mitigate codebook collapse, these methods primarily focus on balancing codebook usage rather than directly reducing quantization errors, leaving the problem unresolved. Summary of the Invention

[0005] Purpose of the Invention: To address the limitations of current methods, this invention proposes a lightweight and high-performance neural speech codec designed to improve feature representation. It includes two main innovations: 1) a causal frequency-domain encoder-decoder equipped with an Inception Residual Block (IRB), which enhances the utilization of multi-scale speech correlations; and 2) a novel Adjustable Residual Vector Quantization (R-RVQ) module that adaptively adjusts residuals to a quantizable form across multiple stages, thereby improving quantization efficiency.

[0006] This invention first provides a lightweight streaming speech coding system based on neural networks, including an encoding compression end and a decoding reconstruction end;

[0007] The encoding and compression end includes a speech encoder and a quantizer;

[0008] The speech encoder includes a time-frequency transformation module, a feature extraction module, an encoding end channel transformation module, and a long-range time-domain correlation extraction module;

[0009] The feature extraction module includes a time-frequency feature extraction submodule and a time-frequency size downsampling submodule;

[0010] The quantizer includes two or more quantization modules, and each quantization module includes a first domain transformation module at the encoding end, a vector quantization module at the encoding end, an inverse vector quantization module at the encoding end, and a second domain transformation module at the encoding end.

[0011] The decoding and reconstruction end includes an inverse quantizer and a speech decoder;

[0012] The inverse quantizer includes two or more layers of inverse quantization modules, each layer of inverse quantization module including a decoding-end inverse vector quantization module and a decoding-end first domain transformation module;

[0013] The inverse vector quantization module and the first domain transformation module at the decoding end of each inverse quantization module correspond to the inverse vector quantization module and the second domain transformation module at the encoding end of the quantization module of the corresponding layer.

[0014] The speech decoder includes a long-range time-domain correlation analysis module, a decoding-end channel transformation module, a feature reconstruction module, and a time-frequency inverse transformation module;

[0015] The feature reconstruction module includes a time-frequency feature parsing submodule and a time-frequency size upsampling submodule;

[0016] The feature extraction module has N1 components, where N1 is a natural number greater than or equal to 2.

[0017] There are N1 feature reconstruction modules.

[0018] In the speech encoder, the time-frequency transformation module transforms the time-domain audio signal into time-frequency features through time-frequency transformation;

[0019] The time-frequency feature extraction submodule is used to extract the local correlation between the time domain dimension and the frequency domain dimension in time-frequency features at multiple scales;

[0020] The time-frequency size downsampling submodule downsamples the time-frequency features, compressing them into a more compact representation.

[0021] Stacking two or more feature extraction modules allows time-frequency features to be gradually extracted from shallow features into more compact deep features;

[0022] The encoding end channel transformation module is used to adjust the number of channels of time-frequency features, thereby controlling the number of model parameters and computational load;

[0023] The long-range temporal correlation extraction module is used to extract the long-range correlation of the temporal dimension in deep features.

[0024] In the quantizer, the quantization module performs layer-by-layer quantization on the deep time-frequency features output by the speech encoder. After the current layer quantization module quantizes the features, the quantization residual is sent to the next layer and quantized by the next layer quantization module.

[0025] In each quantization module, the first domain transformation module at the encoder performs a domain transformation on the input features of the current layer. Then, the vector quantization module at the encoder quantizes the features into a sequence of numbers. The sequence of numbers is interpreted into quantized features by the inverse vector quantization module at the encoder. The quantized features are then transformed into the output features of the current layer by the second domain transformation module at the encoder. The residual between the input features of the current layer and the output features of the current layer is calculated, and the resulting residual is used as the input features of the next layer.

[0026] The final output of the quantizer is a quantized output sequence formed by combining the sequence numbers output by the vector quantization modules at the encoding end of each quantization module.

[0027] In the inverse quantizer, the inverse quantization module performs hierarchical inverse quantization on the quantized output sequence number. After each layer of inverse quantization module performs inverse quantization, the inverse quantization output features of all layers are added together to obtain the final quantized reconstructed features. In each layer of inverse quantization module, the sequence number is decoded into quantized features by the inverse vector quantization module at the decoding end, and the quantized features are transformed into the inverse quantization output features of the current layer by the first domain transformation module at the decoding end.

[0028] In the speech decoder, the long-range temporal correlation parsing module is used to parse the long-range correlation of the temporal dimension in the quantized reconstructed features;

[0029] The decoding end channel conversion module is used to adjust the number of channels of the feature, thereby controlling the number of parameters and the amount of computation of the system;

[0030] The time-frequency feature parsing submodule is used to parse the multi-scale local correlations between the time domain dimension and the frequency domain dimension in time-frequency features;

[0031] The time-frequency size upsampling submodule upsamples the time-frequency features;

[0032] Stacking two or more feature reconstruction modules allows time-frequency features to be gradually reconstructed from deep features into features used for reconstructing shallow features.

[0033] The inverse time-frequency transformation module transforms shallow time-frequency features into reconstructed time-domain audio signals through inverse time-frequency transformation.

[0034] The present invention also provides a lightweight streaming speech coding method based on neural networks implemented using the aforementioned system, comprising the following steps:

[0035] Step 1, streaming encoding compression, includes the following steps:

[0036] Step 1.1: Divide the test audio into short time frames and stream them into the system sequentially;

[0037] Step 1.2: Each small frame is encoded and compressed by the speech encoder into a deep time-frequency feature representation;

[0038] Step 1.3: The deep feature representation is quantized into a sequence of numbers by the quantizer, and output as a bitstream in sequence.

[0039] Step 2, streaming decoding and reconstruction, includes the following steps:

[0040] Step 2.1: Receive the sequence number code stream sequentially, input it into the inverse quantizer, and decode the quantization reconstruction features of the small frames;

[0041] Step 2.2: The quantized reconstruction features of each small frame are decoded into audio signals of the small frame by the speech decoder;

[0042] Step 2.3: The audio signals of small frames are streamed and gradually spliced ​​together to reconstruct the output audio.

[0043] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0044] The present invention also provides a storage medium storing a computer program or instructions that, when the computer program or instructions are run on a computer, execute the steps of the method described.

[0045] This invention offers the following advantages: The method uses a neural network for speech coding, effectively extracting multi-scale feature information from the speech signal, enabling high-quality speech compression coding and reconstruction at extremely low bit rates. This invention has a small number of parameters, low complexity, and can achieve streaming encoding and decoding. Attached Figure Description

[0046] Figure 1 This is a schematic diagram of a lightweight streaming speech codec structure based on a neural network, provided by the present invention.

[0047] Figure 2 This is a schematic diagram of the time-frequency feature extraction submodule.

[0048] Figure 3 This is a schematic diagram of a domain transformation submodule.

[0049] Figure 4 This is a schematic diagram of a lightweight streaming speech codec based on neural networks. Detailed Implementation

[0050] like Figure 1 As shown, this embodiment of the invention provides a lightweight streaming speech coding system based on neural networks, including an encoding compression end and a decoding reconstruction end; the encoding compression end includes a speech encoder and a quantizer.

[0051] The decoding and reconstruction end includes an inverse quantizer and a speech decoder.

[0052] In speech encoders, the short-time Fourier transform is used as the time-frequency transform. For example... Figure 2 The structure shown serves as a time-frequency feature extraction submodule, using causal convolution with stride as a time-frequency downsampling submodule, combined to form a feature extraction module, for a total of four feature extraction modules. Linear layers are used as the encoding-end channel transformation module, and a long short-term memory network is used as the long-range temporal correlation extraction module. For example... Figure 1 As shown, the quantizer uses a total of 6 layers of quantization modules stacked together. Figure 1 In this context, y0 represents the deep time-frequency features output by the speech encoder, and r0 represents the first-layer input features of the quantizer. r1 represents the output features of the first layer, and r1 represents the input features of the second layer of the quantizer. This represents the output features of the second layer. This represents the quantitative reconstruction characteristics.

[0053] Use such as Figure 3 The domain transformation submodules shown serve as the first domain transformation module and the second domain transformation module at the encoding end, and use single vector quantization as the vector quantization module and the inverse vector quantization module.

[0054] On the decoding side, the inverse quantizer uses a stack of six layers of inverse quantization modules. A single vector quantization is used as the inverse vector quantization module, employing methods such as... Figure 3 The domain transformation submodule shown serves as the first domain transformation module at the decoding end. In the speech decoder, a Long Short-Term Memory (LSTM) network is used as the long-range temporal correlation resolution module, and a linear layer is used as the channel transformation module at the decoding end. (The last sentence appears to be incomplete and possibly refers to a different module.) Figure 2 The structure shown serves as a time-frequency feature parsing submodule, using a causal transpose convolution with stride as a time-frequency upsampling submodule, which together form a feature reconstruction module, for a total of four feature reconstruction modules. Inverse short-time Fourier transform is used as the time-frequency inverse transform module.

[0055] This invention also provides a lightweight streaming speech coding method based on neural networks implemented using the system described above. In this embodiment, the test-clean subset of the LibriTTS dataset is used for streaming encoding and decoding tests.

[0056] When performing actual streaming codec tests, such as... Figure 4 As shown, perform the following steps:

[0057] Step 1, streaming encoding compression, includes the following steps:

[0058] Step 1.1: Divide the test audio into short frames, each frame lasting 80ms, and input them sequentially into the streaming model;

[0059] Step 1.2: Each small frame is encoded and compressed by the speech encoder into a deep time-frequency feature representation;

[0060] Step 1.3: The deep feature representation is quantized into a sequence of numbers by the quantizer, and output as a bitstream in sequence.

[0061] Step 2, streaming decoding and reconstruction, includes the following steps:

[0062] Step 2.1: Receive the sequence number code stream sequentially, input it into the inverse quantizer, and decode the quantization reconstruction features of the small frames;

[0063] Step 2.2: The quantized reconstruction features of each small frame are decoded into audio signals of the small frame by the speech decoder;

[0064] Step 2.3: The audio signals of small frames are streamed and gradually spliced ​​together to reconstruct the output audio.

[0065] Table 1 shows a comparison of the test results of this system with those of other existing methods.

[0066] Table 1 Comparison of Results

[0067]

[0068] ViSQOL is the objective audience evaluation index for virtual speech quality, STOI is short-term objective intelligibility, and PESQ is the perceptual evaluation index for speech quality. The higher the value of each of the three indices, the better the speech reconstruction effect.

[0069] This invention also provides a computer program product, including computer instructions stored in a computer-readable storage medium. When a processor of an electronic device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, causing the electronic device to perform the steps of the method.

[0070] This invention provides a lightweight streaming speech coding system and method based on neural networks. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A neural network-based lightweight streaming speech coding system, characterized by, The coding compression end and the decoding reconstruction end are included; The coding compression end comprises a speech encoder and a quantizer; The speech encoder comprises a time-frequency transform module, a feature extraction module, an encoding end channel transform module, and a long-range time-domain correlation extraction module; The feature extraction module comprises a time-frequency feature extraction submodule and a time-frequency size downsampling submodule; The quantizer comprises two or more layers of quantization modules, each layer of quantization module comprising an encoding end first domain transform module, an encoding end vector quantization module, an encoding end inverse vector quantization module, and an encoding end second domain transform module; The decoding reconstruction end comprises an inverse quantizer and a speech decoder; The inverse quantizer comprises two or more layers of inverse quantization modules, each layer of inverse quantization module comprising a decoding end inverse vector quantization module and a decoding end first domain transform module; The decoding end inverse vector quantization module and the decoding end first domain transform module in each layer of inverse quantization module correspond to the encoding end inverse vector quantization module and the encoding end second domain transform module in the corresponding layer of quantization module; The speech decoder comprises a long-range time-domain correlation analysis module, a decoding end channel transform module, a feature reconstruction module, and a time-frequency inverse transform module; The feature reconstruction module comprises a time-frequency feature analysis submodule and a time-frequency size upsampling submodule.

2. The system of claim 1, wherein, The feature extraction module has N1, and N1 is a natural number greater than or equal to 2.

3. The system of claim 2, wherein, The feature reconstruction module has N1.

4. The system of claim 3, wherein, In the speech encoder, the time-frequency transform module converts a time-domain audio signal into a time-frequency feature through time-frequency transform; The time-frequency feature extraction submodule is used to extract the local correlation of the time-domain dimension and the frequency-domain dimension in the time-frequency feature in multiple scales; The time-frequency size downsampling submodule downsamples the time-frequency feature to compress the time-frequency feature into a more compact expression; Stacking two or more feature extraction modules enables the time-frequency feature to be gradually extracted from shallow features to more compact deep features; The encoding end channel transform module is used to adjust the number of channels of the time-frequency feature, thereby controlling the parameter quantity and the calculation quantity of the model; The long-range time-domain correlation extraction module is used to extract the long-range correlation of the time-domain dimension in the deep feature.

5. The system of claim 4, wherein, In the quantizer, the quantization module quantizes the deep time-frequency feature output by the speech encoder layer by layer, and after the feature is quantized by the current layer quantization module, the quantization residual is sent to the next layer for quantization by the quantization module of the next layer; In each layer of quantization module, the encoding end first domain transform module performs domain transformation on the input feature of the current layer, and then the encoding end vector quantization module quantizes the feature into a sequence number sequence, which is interpreted into a quantized feature by the encoding end inverse vector quantization module, and the quantized feature is converted into the output feature of the current layer by the encoding end second domain transform module; the residual obtained by subtracting the input feature of the current layer from the output feature of the current layer is used as the input feature of the next layer; The quantizer finally outputs a quantization output sequence number sequence composed of the sequence number sequences output by the encoding end vector quantization modules in each layer of quantization module.

6. The system of claim 5, wherein, In the inverse quantizer, the inverse quantization modules perform hierarchical inverse quantization on the quantized output index sequence, and the inverse quantization output features of all layers are added to obtain the final quantized reconstruction features; in each inverse quantization module, the index sequence is interpreted into quantized features by the decoding end inverse vector quantization module, and the quantized features are converted into the current layer inverse quantization output features by the decoding end first domain transformation module.

7. The system of claim 6, wherein, In the speech decoder, the long-range time domain correlation analysis module is used to analyze the long-range correlation of the time domain dimension in the quantized reconstruction features. The decoding end channel transformation module is used to adjust the number of channels of the features, thereby controlling the parameter quantity and calculation quantity of the system. The time-frequency feature analysis submodule is used to analyze the multi-scale local correlation of the time domain dimension and the frequency domain dimension in the time-frequency features. The time-frequency dimension upsampling submodule upsamples the time-frequency features. The stacking of two or more feature reconstruction modules enables the time-frequency features to be gradually reconstructed from deep features to shallow features. The time-frequency inverse transformation module converts the shallow time-frequency features into reconstructed time-domain audio signals through time-frequency inverse transformation.

8. A neural network based light-weight streaming speech coding method implemented using the system of any one of claims 1-7. The method comprises the following steps: Step 1, streaming encoding compression, comprising the following steps: Step 1.1, the test audio is divided into short time small frames, which are sequentially input into the system in a streaming manner; Step 1.2, each small frame is encoded and compressed into a deep time-frequency feature representation by a speech encoder; Step 1.3, the deep feature representation is quantized into an index sequence by a quantizer, which is output as a code stream in a sequential streaming manner; Step 2, streaming decoding reconstruction, comprising the following steps: Step 2.1, sequentially receive the index sequence code stream, and input it into the inverse quantizer to interpret the quantized reconstruction features of the small frames; Step 2.2, the quantized reconstruction features of each small frame are decoded into audio signals of the small frames by a speech decoder; Step 2.3, the audio signals of the small frames are output in a streaming manner, and are gradually spliced to reconstruct the output audio.

9. An electronic device, comprising: The processor and the memory are included, and the memory stores program codes, which, when executed by the processor, cause the processor to execute the steps of the method of claim 8.

10. A storage medium, characterized by The computer program or instructions are stored, and when the computer program or instructions are run on a computer, the steps of the method of claim 8 are executed.

Images