Lightweight streaming speech coding system and method based on neural network

Abstract

The invention provides a lightweight streaming speech coding system and method based on a neural network. The system comprises a coding compression end and a decoding reconstruction end, the coding compression end comprises a voice coder and a quantizer; the voice encoder comprises a time-frequency transformation module, a feature extraction module, an encoding end channel transformation module and a long-range time domain correlation extraction module. The feature extraction module comprises a time-frequency feature extraction sub-module and a time-frequency size sampling sub-module; the quantizer comprises more than two layers of quantization modules, and each layer of quantization module comprises a coding end first domain transformation module, a coding end vector quantization module, a coding end reverse quantity quantization module and a coding end second domain transformation module; and the decoding reconstruction end comprises an inverse quantizer and a voice decoder. According to the method, the multi-scale feature information in the voice signal can be effectively extracted, so that high-quality voice compression coding and reconstruction can be realized at an extremely low code rate, the parameter quantity is small, the complexity is low, and streaming coding and decoding can be realized.

Description

Technical Field

[0001] This invention relates to the field of speech coding, and in particular to a lightweight streaming speech coding system and method based on neural networks. Background Technology

[0002] For a long time, speech coding has been the cornerstone of digital signal processing, enabling efficient transmission and storage of speech signals in real-time communication systems. Traditional speech codecs perform well at medium bit rates, but their performance drops sharply at low and ultra-low bit rates, which severely limits applications such as web conferencing, cloud gaming, and interactive communication.

[0003] In recent years, advancements in neural speech coding technology have yielded significant results in addressing these challenges. By jointly learning compact latent representations through an encoder-quantizer-decoder architecture, neural speech codecs have substantially improved the perceptual quality of encoded speech, particularly in low bitrate scenarios. Among various quantization techniques, residual vector quantization (RVQ) has become the mainstream technique for discretizing latent spaces because it can effectively capture multi-stage structured residual information.

[0004] However, two key challenges remain. First, many encoder-decoder backbones still lack the ability to adequately model multi-scale and contextual speech structures, limiting their ability to extract compact representations and thus constraining rate-distortion performance. Second, standard RVQ processes often face difficulties after the first stage: residuals become irregular and misaligned with codebook vectors, leading to increased quantization loss, inefficient codebook utilization, and potential representation redundancy. Although recent research has proposed techniques to improve codebook utilization and mitigate codebook collapse, these methods primarily focus on balancing codebook usage rather than directly reducing quantization errors, leaving the problem unresolved. Summary of the Invention

[0005] Purpose of the Invention: To address the limitations of current methods, this invention proposes a lightweight and high-performance neural speech codec designed to improve feature representation. It includes two main innovations: 1) a causal frequency-domain encoder-decoder equipped with an Inception Residual Block (IRB), which enhances the utilization of multi-scale speech correlations; and 2) a novel Adjustable Residual Vector Quantization (R-RVQ) module that adaptively adjusts residuals to a quantizable form across multiple stages, thereby improving quantization efficiency.

[0006] This invention first provides a lightweight streaming speech coding system based on neural networks, including an encoding compression end and a decoding reconstruction end; The encoding and compression end includes a speech encoder and a quantizer; The speech encoder includes a time-frequency transformation module, a feature extraction module, an encoding end channel transformation module, and a long-range time-domain correlation extraction module; The feature extraction module includes a time-frequency feature extraction submodule and a time-frequency size downsampling submodule; The quantizer includes two or more quantization modules, and each quantization module includes a first domain transformation module at the encoding end, a vector quantization module at the encoding end, an inverse vector quantization module at the encoding end, and a second domain transformation module at the encoding end. The decoding and reconstruction end includes an inverse quantizer and a speech decoder; The inverse quantizer includes two or more layers of inverse quantization modules, each layer of inverse quantization module including a decoding-end inverse vector quantization module and a decoding-end first domain transformation module; The inverse vector quantization module and the first domain transformation module at the decoding end of each inverse quantization module correspond to the inverse vector quantization module and the second domain transformation module at the encoding end of the quantization module of the corresponding layer. The speech decoder includes a long-range time-domain correlation analysis module, a decoding-end channel transformation module, a feature reconstruction module, and a time-frequency inverse transformation module; The feature reconstruction module includes a time-frequency feature parsing submodule and a time-frequency size upsampling submodule; The feature extraction module has N1 components, where N1 is a natural number greater than or equal to 2.

[0007] There are N1 feature reconstruction modules.

[0008] In the speech encoder, the time-frequency transformation module transforms the time-domain audio signal into time-frequency features through time-frequency transformation; The time-frequency feature extraction submodule is used to extract the local correlation between the time domain dimension and the frequency domain dimension in time-frequency features at multiple scales; The time-frequency size downsampling submodule downsamples the time-frequency features, compressing them into a more compact representation. Stacking two or more feature extraction modules allows time-frequency features to be gradually extracted from shallow features into more compact deep features; The encoding end channel transformation module is used to adjust the number of channels of time-frequency features, thereby controlling the number of model parameters and computational load; The long-range temporal correlation extraction module is used to extract the long-range correlation of the temporal dimension in deep features.

[0009] In the quantizer, the quantization module performs layer-by-layer quantization on the deep time-frequency features output by the speech encoder. After the current layer quantization module quantizes the features, the quantization residual is sent to the next layer and quantized by the next layer quantization module. In each quantization module, the first domain transformation module at the encoder performs a domain transformation on the input features of the current layer. Then, the vector quantization module at the encoder quantizes the features into a sequence of numbers. The sequence of numbers is interpreted into quantized features by the inverse vector quantization module at the encoder. The quantized features are then transformed into the output features of the current layer by the second domain transformation module at the encoder. The residual between the input features of the current layer and the output features of the current layer is calculated, and the resulting residual is used as the input features of the next layer. The final output of the quantizer is a quantized output sequence formed by combining the sequence numbers output by the vector quantization modules at the encoding end of each quantization module.

[0010] In the inverse quantizer, the inverse quantization module performs hierarchical inverse quantization on the quantized output sequence number. After each layer of inverse quantization module performs inverse quantization, the inverse quantization output features of all layers are added together to obtain the final quantized reconstructed features. In each layer of inverse quantization module, the sequence number is decoded into quantized features by the inverse vector quantization module at the decoding end, and the quantized features are transformed into the inverse quantization output features of the current layer by the first domain transformation module at the decoding end.

[0011] In the speech decoder, the long-range temporal correlation parsing module is used to parse the long-range correlation of the temporal dimension in the quantized reconstructed features; The decoding end channel conversion module is used to adjust the number of channels of the feature, thereby controlling the number of parameters and the amount of computation of the system; The time-frequency feature parsing submodule is used to parse the multi-scale local correlations between the time domain dimension and the frequency domain dimension in time-frequency features; The time-frequency size upsampling submodule upsamples the time-frequency features; Stacking two or more feature reconstruction modules allows time-frequency features to be gradually reconstructed from deep features into features used for reconstructing shallow features. The inverse time-frequency transformation module transforms shallow time-frequency features into reconstructed time-domain audio signals through inverse time-frequency transformation.

[0012] The present invention also provides a lightweight streaming speech coding method based on neural networks implemented using the aforementioned system, comprising the following steps: Step 1, streaming encoding compression, includes the following steps: Step 1.1: Divide the test audio into short time frames and stream them into the system sequentially; Step 1.2: Each small frame is encoded and compressed by the speech encoder into a deep time-frequency feature representation; Step 1.3: The deep feature representation is quantized into a sequence of numbers by the quantizer, and output as a bitstream in sequence. Step 2, streaming decoding and reconstruction, includes the following steps: Step 2.1: Receive the sequence number code stream sequentially, input it into the inverse quantizer, and decode the quantization reconstruction features of the small frames; Step 2.2: The quantized reconstruction features of each small frame are decoded into audio signals of the small frame by the speech decoder; Step 2.3: The audio signals of small frames are streamed and gradually spliced ​​together to reconstruct the output audio.

[0013] The present invention also provides an electronic device, including a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method.

[0014] The present invention also provides a storage medium storing a computer program or instructions that, when the computer program or instructions are run on a computer, execute the steps of the method described.

[0015] This invention offers the following advantages: The method uses a neural network for speech coding, effectively extracting multi-scale feature information from the speech signal, enabling high-quality speech compression coding and reconstruction at extremely low bit rates. This invention has a small number of parameters, low complexity, and can achieve streaming encoding and decoding. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of a lightweight streaming speech codec structure based on a neural network, provided by the present invention.

[0017] Figure 2 This is a schematic diagram of the time-frequency feature extraction submodule.

[0018] Figure 3 This is a schematic diagram of a domain transformation submodule.

[0019] Figure 4 This is a schematic diagram of a lightweight streaming speech codec based on neural networks. Detailed Implementation

[0020] like Figure 1 As shown, this embodiment of the invention provides a lightweight streaming speech coding system based on neural networks, including an encoding compression end and a decoding reconstruction end; the encoding compression end includes a speech encoder and a quantizer.

[0021] The decoding and reconstruction end includes an inverse quantizer and a speech decoder.

[0022] In speech encoders, the short-time Fourier transform is used as the time-frequency transform. For example... Figure 2 The structure shown serves as a time-frequency feature extraction submodule, using causal convolution with stride as a time-frequency downsampling submodule, combined to form a feature extraction module, for a total of four feature extraction modules. Linear layers are used as the encoding-end channel transformation module, and a long short-term memory network is used as the long-range temporal correlation extraction module. For example... Figure 1 As shown, the quantizer uses a total of 6 layers of quantization modules stacked together. Figure 1 In this context, y0 represents the deep time-frequency features output by the speech encoder, and r0 represents the first-layer input features of the quantizer. r1 represents the output features of the first layer, and r1 represents the input features of the second layer of the quantizer. This represents the output features of the second layer. This represents the quantitative reconstruction characteristics.

[0023] Use such as Figure 3 The domain transformation submodules shown serve as the first domain transformation module and the second domain transformation module at the encoding end, and use single vector quantization as the vector quantization module and the inverse vector quantization module.

[0024] On the decoding side, the inverse quantizer uses a stack of six layers of inverse quantization modules. A single vector quantization is used as the inverse vector quantization module, employing methods such as... Figure 3 The domain transformation submodule shown serves as the first domain transformation module at the decoding end. In the speech decoder, a Long Short-Term Memory (LSTM) network is used as the long-range temporal correlation resolution module, and a linear layer is used as the channel transformation module at the decoding end. (The last sentence appears to be incomplete and possibly refers to a different module.) Figure 2 The structure shown serves as a time-frequency feature parsing submodule, using a causal transpose convolution with stride as a time-frequency upsampling submodule, which together form a feature reconstruction module, for a total of four feature reconstruction modules. Inverse short-time Fourier transform is used as the time-frequency inverse transform module.

[0025] This invention also provides a lightweight streaming speech coding method based on neural networks implemented using the system described above. In this embodiment, the test-clean subset of the LibriTTS dataset is used for streaming encoding and decoding tests.

[0026] When performing actual streaming codec tests, such as... Figure 4 As shown, perform the following steps: Step 1, streaming encoding compression, includes the following steps: Step 1.1: Divide the test audio into short frames, each frame lasting 80ms, and input them sequentially into the streaming model; Step 1.2: Each small frame is encoded and compressed by the speech encoder into a deep time-frequency feature representation; Step 1.3: The deep feature representation is quantized into a sequence of numbers by the quantizer, and output as a bitstream in sequence. Step 2, streaming decoding and reconstruction, includes the following steps: Step 2.1: Receive the sequence number code stream sequentially, input it into the inverse quantizer, and decode the quantization reconstruction features of the small frames; Step 2.2: The quantized reconstruction features of each small frame are decoded into audio signals of the small frame by the speech decoder; Step 2.3: The audio signals of small frames are streamed and gradually spliced ​​together to reconstruct the output audio.

[0027] Table 1 shows a comparison of the test results of this system with those of other existing methods.

[0028] Table 1 Comparison of Results

[0029] ViSQOL is the objective audience evaluation index for virtual speech quality, STOI is short-term objective intelligibility, and PESQ is the perceptual evaluation index for speech quality. The higher the value of each of the three indices, the better the speech reconstruction effect.

[0030] This invention also provides a computer program product, including computer instructions stored in a computer-readable storage medium. When a processor of an electronic device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, causing the electronic device to perform the steps of the method.

[0031] This invention provides a lightweight streaming speech coding system and method based on neural networks. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A lightweight streaming speech coding system based on neural networks, characterized in that, Includes the encoding / compression end and the decoding / reconstruction end; The encoding and compression end includes a speech encoder and a quantizer; The speech encoder includes a time-frequency transformation module, a feature extraction module, an encoding end channel transformation module, and a long-range time-domain correlation extraction module; The feature extraction module includes a time-frequency feature extraction submodule and a time-frequency size downsampling submodule; The quantizer includes two or more quantization modules, and each quantization module includes a first domain transformation module at the encoding end, a vector quantization module at the encoding end, an inverse vector quantization module at the encoding end, and a second domain transformation module at the encoding end. The decoding and reconstruction end includes an inverse quantizer and a speech decoder; The inverse quantizer includes two or more layers of inverse quantization modules, each layer of inverse quantization module including a decoding-end inverse vector quantization module and a decoding-end first domain transformation module; The inverse vector quantization module and the first domain transformation module at the decoding end of each inverse quantization module correspond to the inverse vector quantization module and the second domain transformation module at the encoding end of the quantization module of the corresponding layer. The speech decoder includes a long-range time-domain correlation analysis module, a decoding-end channel transformation module, a feature reconstruction module, and a time-frequency inverse transformation module; The feature reconstruction module includes a time-frequency feature parsing submodule and a time-frequency size upsampling submodule.

2. The system as described in claim 1, characterized in that, The feature extraction module has N1 components, where N1 is a natural number greater than or equal to 2.

3. The system as described in claim 2, characterized in that, There are N1 feature reconstruction modules.

4. The system as described in claim 3, characterized in that, In the speech encoder, the time-frequency transformation module transforms the time-domain audio signal into time-frequency features through time-frequency transformation; The time-frequency feature extraction submodule is used to extract the local correlation between the time domain dimension and the frequency domain dimension in time-frequency features at multiple scales; The time-frequency size downsampling submodule downsamples the time-frequency features, compressing them into a more compact representation. Stacking two or more feature extraction modules allows time-frequency features to be gradually extracted from shallow features into more compact deep features; The encoding end channel transformation module is used to adjust the number of channels of time-frequency features, thereby controlling the number of model parameters and computational load; The long-range temporal correlation extraction module is used to extract the long-range correlation of the temporal dimension in deep features.

5. The system as described in claim 4, characterized in that, In the quantizer, the quantization module performs layer-by-layer quantization on the deep time-frequency features output by the speech encoder. After the current layer quantization module quantizes the features, the quantization residual is sent to the next layer and quantized by the next layer quantization module. In each quantization module, the first domain transformation module at the encoder performs a domain transformation on the input features of the current layer. Then, the vector quantization module at the encoder quantizes the features into a sequence of numbers. The sequence of numbers is interpreted into quantized features by the inverse vector quantization module at the encoder. The quantized features are then transformed into the output features of the current layer by the second domain transformation module at the encoder. The residual between the input features of the current layer and the output features of the current layer is calculated, and the resulting residual is used as the input features of the next layer. The final output of the quantizer is a quantized output sequence formed by combining the sequence numbers output by the vector quantization modules at the encoding end of each quantization module.

6. The system as described in claim 5, characterized in that, In the inverse quantizer, the inverse quantization module performs hierarchical inverse quantization on the quantized output sequence number. After each layer of inverse quantization module performs inverse quantization, the inverse quantization output features of all layers are added together to obtain the final quantized reconstructed features. In each layer of inverse quantization module, the sequence number is decoded into quantized features by the inverse vector quantization module at the decoding end, and the quantized features are transformed into the inverse quantization output features of the current layer by the first domain transformation module at the decoding end.

7. The system as described in claim 6, characterized in that, In the speech decoder, the long-range temporal correlation parsing module is used to parse the long-range correlation of the temporal dimension in the quantized reconstructed features; The decoding end channel conversion module is used to adjust the number of channels of the feature, thereby controlling the number of parameters and the amount of computation of the system; The time-frequency feature parsing submodule is used to parse the multi-scale local correlations between the time domain dimension and the frequency domain dimension in time-frequency features; The time-frequency size upsampling submodule upsamples the time-frequency features; Stacking two or more feature reconstruction modules allows time-frequency features to be gradually reconstructed from deep features into features used for reconstructing shallow features. The inverse time-frequency transformation module transforms shallow time-frequency features into reconstructed time-domain audio signals through inverse time-frequency transformation.

8. A lightweight streaming speech coding method based on neural networks, implemented using the system described in any one of claims 1 to 7, characterized in that, Includes the following steps: Step 1, streaming encoding compression, includes the following steps: Step 1.1: Divide the test audio into short time frames and stream them into the system sequentially; Step 1.2: Each small frame is encoded and compressed by the speech encoder into a deep time-frequency feature representation; Step 1.3: The deep feature representation is quantized into a sequence of numbers by the quantizer, and output as a bitstream in sequence. Step 2, streaming decoding and reconstruction, includes the following steps: Step 2.1: Receive the sequence number code stream sequentially, input it into the inverse quantizer, and decode the quantization reconstruction features of the small frames; Step 2.2: The quantized reconstruction features of each small frame are decoded into audio signals of the small frame by the speech decoder; Step 2.3: The audio signals of small frames are streamed and gradually spliced ​​together to reconstruct the output audio.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program code that, when executed by the processor, causes the processor to perform the steps of the method as described in claim 8.

10. A storage medium, characterized in that, It stores a computer program or instructions that, when run on a computer, perform the steps of the method as described in claim 8.

Images