A method and system for super low bit-rate audio communication based on vector quantization variational autoencoder

Abstract

The application provides a kind of based on vector quantization variational autoencoder ultra-low code rate audio communication method and system, the steps of the method include encoding processing and decoding processing, the steps of the encoding processing include: extracting the Bark frequency cepstral coefficient feature in each frame of speech data, the Bark frequency cepstral coefficient feature of continuous multiple frames is constructed as a two-dimensional time-frequency feature block;Two-dimensional time-frequency feature block is input to the preset two-dimensional convolutional encoder, and the two-dimensional convolutional encoder outputs continuous latent representation;The continuous latent representation is input to the preset TAEM module, and the enhanced latent representation is obtained by the TAEM module;Based on the enhanced latent representation and the preset codebook, an output code stream is constructed;The steps of the decoding processing include: based on the output code stream, obtain the quantized latent representation, input to the two-dimensional convolutional decoder, obtain the Bark frequency cepstral coefficient feature block, and input to the preset TSPN module, to obtain the reconstructed Bark frequency cepstral coefficient parameter sequence.

Description

Technical Field

[0001] This invention relates to the field of speech coding technology, and in particular to an ultra-low bit rate audio communication method and system based on vector quantization variational autoencoder. Background Technology

[0002] Ultra-low bitrate speech coding is a core technology in satellite communication, emergency communication, and underwater communication. It enables voice information transmission under extremely narrow bandwidth conditions and has advantages such as low communication cost, good concealment, and low terminal power consumption. To maintain acceptable speech quality at limited bitrates, modern low bitrate vocoders, such as Mixed Excitation Linear Prediction Enhanced (MELPe) vocoders and Linear Predictive Coding Networks (LPCNet), typically employ parametric coding frameworks to efficiently compress parameters such as speech spectral envelope, pitch period, and voiced / unvoiced markers. Among these, the spectral envelope parameter directly determines the clarity and naturalness of the reconstructed speech, and its quantization accuracy and temporal stability are key bottlenecks in vocoder performance.

[0003] Neural vocoder systems typically employ multi-level vector quantization to encode speech spectral envelope parameters, such as Mel-frequency cepstral coefficients (MFCCs) and BFCCs. BFCC has unique advantages: it is designed based on the Bark auditory scale, better aligning with human hearing characteristics, and is naturally suited for bandwidth expansion tasks. It can more efficiently reconstruct high-frequency components using limited bandwidth at ultra-low bitrates, thus improving the naturalness of the speech. However, when the bitrate is further reduced to below 1 kbps, the reconstruction error of multi-level vector quantization increases sharply, and the continuity between parameter frames is poor, resulting in a "metallic" or discontinuous feel in the synthesized speech. The root cause is that existing technologies typically use multi-level vector quantization to directly quantize the BFCC. However, the BFCC, as a feature parameter, was not originally designed to serve quantization scenarios, so direct quantization struggles to achieve optimal quantization efficiency or accuracy. Furthermore, multi-level residual quantization suffers from error propagation and accumulation, and frame-by-frame independent quantization fails to effectively utilize the temporal correlation between adjacent frames. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an ultra-low bit rate audio communication method and system based on vector quantization variational autoencoder, in order to eliminate or improve one or more defects existing in the prior art.

[0005] One aspect of the present invention provides an ultra-low bit rate audio communication method based on a vector quantization variational autoencoder, the method comprising encoding processing and decoding processing, wherein the encoding processing includes: Extract the Barker frequency cepstral coefficient features from each frame of speech data, and construct a two-dimensional time-frequency feature block from the Barker frequency cepstral coefficient features of multiple consecutive frames of speech data. The two-dimensional time-frequency feature block is input into a preset two-dimensional convolutional encoder, which outputs a continuous latent representation. The two-dimensional convolutional encoder includes multiple convolutional layers. The continuous latent representation is input into a preset TAEM module, and an enhanced latent representation is obtained through the TAEM module. The TAEM module adopts a self-attention mechanism for global context enhancement. The output bitstream is constructed based on the enhanced latent representation and the preset codebook; The decoding process includes the following steps: The output bitstream is obtained, a quantized latent representation is obtained based on the output bitstream, and the quantized latent representation is input into a two-dimensional convolutional decoder to obtain a Barker frequency cepstral coefficient feature block. The Buck frequency cepstral coefficient feature block is input into a preset TSPN module, which includes multiple one-dimensional convolutional layers. The TSPN module is used to perform inter-frame smoothing optimization on the reconstructed parameters in a residual manner to obtain the reconstructed Buck frequency cepstral coefficient parameter sequence.

[0006] The above scheme organizes BFCC parameters into two-dimensional time-frequency feature blocks, allowing one input block to simultaneously cover multiple consecutive speech frames and multiple BFCC feature dimensions. A two-dimensional convolutional encoder extracts joint features in both the time and feature dimensions to generate a continuous latent representation. The TAEM module performs global context enhancement on the spatial location latent vectors in the continuous latent representation, enabling subsequent codeword selection to incorporate the time-frequency correlation within the entire BFCC block. The receiver recovers the quantized latent representation using the same codebook, reconstructs the Barker frequency cepstral coefficient feature block using a two-dimensional convolutional decoder, and then refines the residuals of local time-dimension jumps using a TSPN module temporal smoothing post-processing network. Thus, this scheme ensures the indexed bitstream is suitable for narrowband link transmission while reducing BFCC parameter quantization distortion at ultra-low bit rates, avoiding error propagation and accumulation in multi-level residual quantization, and enhancing the inter-frame temporal correlation of the reconstructed BFCC parameter sequence.

[0007] In some embodiments of the present invention, the decoding process further includes: The reconstructed Buck frequency cepstral coefficient parameter sequence is input into a preset neural vocoder synthesis network to obtain the final reconstructed speech signal.

[0008] In some embodiments of the present invention, in the step of inputting the two-dimensional time-frequency feature block into a preset two-dimensional convolutional encoder, the two-dimensional time-frequency feature block is normalized, and the normalized two-dimensional time-frequency feature block is input into the preset two-dimensional convolutional encoder.

[0009] In some embodiments of the present invention, in the step of inputting the two-dimensional time-frequency feature block into a preset two-dimensional convolutional encoder, the two-dimensional convolutional encoder includes a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, a third two-dimensional convolutional layer, a plurality of two-dimensional residual blocks and a fourth two-dimensional convolutional layer arranged in sequence, and each two-dimensional residual block is provided with two convolutional layers.

[0010] In some embodiments of the present invention, in the step of inputting the continuous latent representation into a preset TAEM module and obtaining an enhanced latent representation through the TAEM module: The two-dimensional time-frequency space of the continuous latent representation is flattened into a feature sequence of a preset length. Each position of the feature sequence corresponds to a first latent vector of a preset number of dimensions. Then, the scaled dot product self-attention is calculated to obtain the attention output. The attention output is residually concatenated with a flattened feature sequence of a preset length to obtain the enhanced latent representation.

[0011] In some embodiments of the present invention, in the step of constructing the output bitstream based on the enhanced latent representation and the preset codebook: For the second latent vector at each position in the enhanced latent representation, calculate the Euclidean distance with each codeword, and take the index of the codeword with the smallest Euclidean distance as the index corresponding to the second latent vector; Combine all the indices corresponding to the second potential vectors and encapsulate them into an output bitstream.

[0012] In some embodiments of the present invention, in the steps of acquiring the output bitstream, obtaining a quantized latent representation based on the output bitstream, and inputting the quantized latent representation into a two-dimensional convolutional decoder to obtain a Barker frequency cepstral coefficient feature block, the corresponding codebook determines the corresponding second latent vector based on the index in the output bitstream to obtain a quantized latent representation, and inputs it into the two-dimensional convolutional decoder. The two-dimensional convolutional decoder includes a first two-dimensional decoding convolutional layer, multiple two-dimensional residual blocks, a first two-dimensional transposed convolutional layer, and a second two-dimensional transposed convolutional layer arranged in sequence to obtain the Barker frequency cepstral coefficient feature block.

[0013] In some embodiments of the present invention, in the step of inputting the Buck frequency cepstral coefficient feature block into a preset TSPN module, the TSPN module including multiple one-dimensional convolutional layers, the TSPN module being used to perform inter-frame smoothing optimization of the reconstructed parameters in a residual manner to obtain the reconstructed Buck frequency cepstral coefficient parameter sequence, the TSPN module includes three one-dimensional convolutional layers arranged in sequence, the kernel sizes of the three one-dimensional convolutional layers being 5, 5 and 3 in sequence.

[0014] In some embodiments of the present invention, in the step of inputting the reconstructed Barker frequency cepstral coefficient parameter sequence into a preset neural vocoder synthesis network to obtain the final reconstructed speech signal, the reconstructed Barker frequency cepstral coefficient parameter sequence is combined with decoding parameters and input into the neural vocoder synthesis network. The decoding parameters include pitch and gain. The neural vocoder synthesis network adopts an LPCNet-like structure or a convolutional recursive hybrid structure.

[0015] A second aspect of the present invention also provides an ultra-low bit rate audio communication system based on a vector quantization variational autoencoder. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0016] A third aspect of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned ultra-low bit rate audio communication method based on vector quantization variational autoencoder.

[0017] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the text, or may be learned by practice of the invention. The objects and other advantages of the invention will become apparent from the description and the accompanying drawings.

[0018] Those skilled in the art will understand that the objectives and advantages achievable with this invention are not limited to those specifically described above, and that the above and other objectives achievable with this invention will become clearer from the following detailed description. Attached Figure Description

[0019] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.

[0020] Figure 1 This is a schematic diagram of one implementation of the ultra-low bit rate audio communication method based on vector quantization variational autoencoder in this scheme; Figure 2 This is a schematic diagram of the overall architecture of this solution; Figure 3 This is a schematic diagram of the architecture for the encoding process in this scheme; Figure 4 This is a schematic diagram of the architecture for decoding processing in this solution; Figure 5 This is a schematic diagram of the experimental results of this scheme. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0022] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0023] like Figure 1 and 2 As shown, this invention proposes an ultra-low bit rate audio communication method based on vector quantization variational autoencoder. The method includes encoding and decoding processes. The encoding process includes the following steps: Step S110: Extract the Barker frequency cepstral coefficient features from each frame of speech data, and construct a two-dimensional time-frequency feature block from the Barker frequency cepstral coefficient features in multiple consecutive frames of speech data. In one optional implementation, dimensions 1 to 17 are extracted from each frame of speech data as Bark Frequency Cepstral Coefficients (BFCC) features, and the target BFCC features of 16 consecutive frames are arranged into a 16×17 two-dimensional time-frequency feature block. The temporal context covered by 16 consecutive frames can be processed simultaneously in two-dimensional convolutional coding and temporal attention enhancement, enabling the spectral envelope variation relationship between adjacent frames to participate in latent representation generation and codeword selection.

[0024] like Figure 3 As shown, Figure 3 The BFCC2D time-frequency block in the model is a two-dimensional time-frequency feature block. In step S120, the two-dimensional time-frequency feature block is input into a preset two-dimensional convolutional encoder. The two-dimensional convolutional encoder outputs a continuous latent representation. The two-dimensional convolutional encoder includes multiple convolutional layers. Specifically, the continuous latent representation is expressed as .

[0025] Step S130: The continuous latent representation is input into a preset TAEM module to obtain an enhanced latent representation. The TAEM module uses a self-attention mechanism for global context enhancement. Step S140: Construct an output bitstream based on the enhanced latent representation and the preset codebook; like Figure 4 As shown, the decoding process includes the following steps: Step S210: Obtain the output bitstream, obtain a quantized latent representation based on the output bitstream, and input the quantized latent representation into a two-dimensional convolutional decoder to obtain a Barker frequency cepstral coefficient feature block; Specifically, the receiving end decapsulates the code to obtain 16 5-bit indices, recovers the corresponding 128-dimensional codewords from the codebook based on the indices, and reassembles them into a quantized latent representation according to their spatial positions. .

[0026] Step S220: Input the Buck frequency cepstral coefficient feature block into a preset TSPN module. The TSPN module includes multiple one-dimensional convolutional layers. The TSPN module is used to perform inter-frame smoothing optimization on the reconstructed parameters in a residual manner to obtain the reconstructed Buck frequency cepstral coefficient parameter sequence.

[0027] Using the above scheme, this scheme first organizes the BFCC parameters into two-dimensional time-frequency feature blocks, so that one input block simultaneously covers multiple consecutive speech frames and multiple BFCC feature dimensions; the two-dimensional convolutional encoder extracts joint features in the time dimension and feature dimension to generate a continuous latent representation; the TAEM module performs global context enhancement on the spatial location latent vectors in the continuous latent representation, so that subsequent codeword selection can combine the time-frequency correlation within the entire BFCC block; the receiver recovers the quantized latent representation using the same codebook, the two-dimensional convolutional decoder reconstructs the Barker frequency cepstral coefficient feature block, and then the TSPN module temporal smoothing post-processing network performs residual refinement on local jumps in the time dimension. Thus, this scheme ensures that the index bitstream is suitable for narrowband link transmission, reduces BFCC parameter quantization distortion at ultra-low bitrates, avoids error propagation and accumulation in multi-level residual quantization, and enhances the inter-frame temporal correlation of the reconstructed BFCC parameter sequence.

[0028] In some embodiments of the present invention, the decoding process further includes: The reconstructed Buck frequency cepstral coefficient parameter sequence is input into a preset neural vocoder synthesis network to obtain the final reconstructed speech signal.

[0029] In some embodiments of the present invention, in the step of inputting the two-dimensional time-frequency feature block into a preset two-dimensional convolutional encoder, the two-dimensional time-frequency feature block is normalized, and the normalized two-dimensional time-frequency feature block is input into the preset two-dimensional convolutional encoder.

[0030] In the specific implementation process, each feature dimension is independently normalized using pre-calculated mean and standard deviation. The normalized two-dimensional time-frequency feature block is then input into a two-dimensional convolutional encoder.

[0031] In some embodiments of the present invention, in the step of inputting the two-dimensional time-frequency feature block into a preset two-dimensional convolutional encoder, the two-dimensional convolutional encoder includes a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, a third two-dimensional convolutional layer, a plurality of two-dimensional residual blocks and a fourth two-dimensional convolutional layer arranged in sequence, and each two-dimensional residual block is provided with two convolutional layers.

[0032] like Figure 3 As shown, in some embodiments of the present invention, the first two-dimensional convolutional layer uses a 4×4 kernel, a 2×2 stride, 1 padding, 1 input channel, 128 output channels, activated by a Gaussian Error Linear Unit (GELU), and has an output size of 8×8×128; the second two-dimensional convolutional layer uses a 4×4 kernel, a 2×2 stride, 1 padding, 128 input channels, 256 output channels, activated by a GELU, and has an output size of 4×4×256; the third two-dimensional convolutional layer uses a 3×3 kernel, a 1×1 stride, 1 padding, 256 input channels, 256 output channels, activated by a GELU, and maintains a size of 4×4×256; multiple two-dimensional residual blocks do not change their spatial dimensions, and the first convolutional layer of the two-dimensional residual block changes channel 256 to channel 128, using a 3×3 kernel, 1 padding, and a Gaussian Error Linear Unit (GELU). The second convolutional layer activates the GELU unit; the second convolutional layer transforms channel 128 into channel 256, using a 1×1 kernel and padding with 0; the fourth two-dimensional convolutional layer uses a 1×1 kernel, a 1×1 stride, padding with 0, input channels 256, output channels 128, no activation function, and output size 128×4×4.

[0033] In some embodiments of the present invention, in the step of inputting the continuous latent representation into a preset TAEM module and obtaining an enhanced latent representation through the TAEM module: The two-dimensional time-frequency space of the continuous latent representation is flattened into a feature sequence of a preset length. Each position of the feature sequence corresponds to a first latent vector of a preset number of dimensions. Then, the scaled dot product self-attention is calculated to obtain the attention output. The attention output is residually concatenated with a flattened feature sequence of a preset length to obtain the enhanced latent representation.

[0034] In its implementation, this scheme considers the strong temporal correlation of BFCC parameters between adjacent frames. While traditional VQ-VAE's two-dimensional convolutional encoder can locally perceive the time-frequency neighborhood, it cannot explicitly model long-range context dependencies. The TAEM module uses a self-attention mechanism to manage the continuous latent representations... Global dependency enhancement is performed on the latent vectors at each spatial location. Specifically, the two-dimensional time-frequency space of the continuous latent representation is flattened into a feature sequence of length 16, with each location corresponding to a 128-dimensional first latent vector. Then, scaled dot product self-attention is calculated. .in , , The query vector, key vector, and value vector are all obtained by linear transformation of the flattened sequence. Let this be the dimension of the query vector. Add the attention output to the original flattened sequence (residual connection), then restore the original sequence. The spatial structure is used to obtain an enhanced latent representation. .

[0035] By adopting the above scheme, the TAEM module enables the subsequent vector quantization process to select codewords under the guidance of the global time-frequency context, which effectively improves the inter-frame continuity and perceptual naturalness of BFCC parameter reconstruction at ultra-low bit rates.

[0036] In some embodiments of the present invention, in the step of constructing the output bitstream based on the enhanced latent representation and the preset codebook: For the second latent vector at each position in the enhanced latent representation, calculate the Euclidean distance with each codeword, and take the index of the codeword with the smallest Euclidean distance as the index corresponding to the second latent vector; Combine all the indices corresponding to the second potential vectors and encapsulate them into an output bitstream.

[0037] In one alternative implementation, the discrete codebook comprises 32 codewords, each codeword being a 128-dimensional vector; the spatial size of the augmented latent representation is 4×4, with each spatial location having a 128-dimensional augmented latent vector. After selecting the nearest codeword for the augmented latent vector at each spatial location, the quantization index for each spatial location is represented by 5 bits, resulting in a total of 16 indices, corresponding to an 80-bit transmission bitstream.

[0038] like Figure 4As shown, in some embodiments of the present invention, in the steps of acquiring the output bitstream, obtaining a quantized latent representation based on the output bitstream, and inputting the quantized latent representation into a two-dimensional convolutional decoder to obtain a Barker frequency cepstral coefficient feature block, the corresponding codebook determines the corresponding second latent vector based on the index in the output bitstream to obtain a quantized latent representation, and inputs it into the two-dimensional convolutional decoder. The two-dimensional convolutional decoder includes a first two-dimensional decoding convolutional layer, multiple two-dimensional residual blocks, a first two-dimensional transposed convolutional layer, and a second two-dimensional transposed convolutional layer arranged in sequence to obtain the Barker frequency cepstral coefficient feature block.

[0039] In some embodiments of the present invention, the first two-dimensional decoding convolutional layer uses a 3×3 kernel, a 1×1 stride, 1 padding, 128 input channels, 256 output channels, GELU activation, and maintains a size of 4×4×256; the first two-dimensional transposed convolutional layer uses a 4×4 kernel, a 2×2 stride, 1 padding, 128 output channels, GELU activation, and upsamples the size from 4×4 to 8×8; the second two-dimensional transposed convolutional layer uses a 4×4 kernel, a 2×2 stride, 1 padding, 1 output channel, no activation function, and upsamples the size from 8×8 to 16×17.

[0040] In some embodiments of the present invention, in the step of inputting the Buck frequency cepstral coefficient feature block into a preset TSPN module, the TSPN module including multiple one-dimensional convolutional layers, the TSPN module being used to perform inter-frame smoothing optimization of the reconstructed parameters in a residual manner to obtain the reconstructed Buck frequency cepstral coefficient parameter sequence, the TSPN module includes three one-dimensional convolutional layers arranged in sequence, the kernel sizes of the three one-dimensional convolutional layers being 5, 5 and 3 in sequence.

[0041] In some embodiments of the present invention, in the step of inputting the reconstructed Barker frequency cepstral coefficient parameter sequence into a preset neural vocoder synthesis network to obtain the final reconstructed speech signal, the reconstructed Barker frequency cepstral coefficient parameter sequence is combined with decoding parameters and input into the neural vocoder synthesis network. The decoding parameters include pitch and gain. The neural vocoder synthesis network adopts an LPCNet-like structure or a convolutional recursive hybrid structure.

[0042] This solution replaces the traditional multi-level vector quantization process, improves the BFCC parameter reconstruction quality while ensuring the transmittability of discrete indexes, reduces quantization distortion under low bit rate conditions, and enhances the temporal continuity of the parameter sequence, thus having clear engineering application value.

[0043] Using the above scheme, the TSPN module of this scheme is designed with the continuity prior of BFCC parameters in the time dimension through a post-processing network. This effectively suppresses possible inter-frame parameter jumps in the decoder output, further reducing the 'metallic' sound and plosive sounds in the synthesized speech. The refined reconstructed BFCC parameter sequence is then inversely normalized using the same mean and standard deviation as the encoder to obtain the final reconstructed BFCC parameter sequence. The final reconstructed BFCC parameter sequence, along with other decoding parameters (pitch and gain, etc.), is fed into the neural vocoder synthesis network to finally reconstruct the speech signal, corresponding to the inverse process of the encoder.

[0044] Example: Input speech: Select a mono audio data segment with a sampling rate of 8kHz and a duration of 10 seconds (content: "The sun rises from the east, the bookstore is still closed"). The speech is first segmented into frames (20 ms long, 50% overlap between frames), and 36-dimensional speech parameters are extracted from each frame, including BFCC coefficients. Dimensions 1 to 17 are extracted from the 36-dimensional parameters of each frame as Barker frequency cepstral coefficient features (17 dimensions in total), and the remaining parameters (pitch and gain, etc.) are encoded and transmitted separately.

[0045] Constructing the BFCC feature block: Organize the Barker frequency cepstral coefficient features of 16 consecutive frames into a two-dimensional time-frequency feature block. The size of the two-dimensional time-frequency feature block is: time dimension: 16 frames (corresponding to 160 ms of speech) and feature dimension: 17 dimensions. The stride of adjacent blocks is equal to the block length (16 frames), that is, there is no overlap between blocks.

[0046] For a 10-second voice recording, the total number of frames is (10s / 0.01s) = 1000 frames, which can be divided into ceil(1000 / 16) = 63 complete blocks of 16 frames each (if the last few frames are less than 16, they are padded with zeros or discarded. In this embodiment, the simple method of discarding the last few frames is used. In actual systems, the overlapping and retention method can be used, which will not be elaborated here).

[0047] Each two-dimensional time-frequency feature block is represented as a tensor. (1 represents the number of channels). The cepstral coefficient features for each Vibak frequency are then normalized using the mean pre-calculated on a large speech database. and standard deviation ,calculate .

[0048] Encoding and Quantization (Transmitter): The first normalized two-dimensional time-frequency feature block (corresponding to the beginning of the speech, 0–160 ms) is input into the encoder. The encoder is a trained two-dimensional convolutional network including a two-dimensional convolutional encoder and a TAEM module, which outputs a continuous latent representation. ,Right now Each position on the grid has a 128-dimensional vector. Subsequently, the TAEM module performs global context augmentation on the latent vectors at 16 spatial locations using a self-attention mechanism, outputting an augmented latent representation. .

[0049] Subsequently Nearest neighbor quantization: codebook It contains 32 128-dimensional codewords. For each of the 16 vectors in the enhanced latent representation, the Euclidean distance between each vector and the 32 codewords is calculated, and the index of the codeword with the smallest distance (0-31) is selected. For example, the vector at the first position may have the smallest distance to the 5th codeword, so the index 5 is output. All 16 positions are indexed sequentially, forming a 4×4 index matrix K.

[0050] Each index value can be represented using 5 bits (because...) =32). Therefore, the input block is compressed into a bitstream of 16 × 5 = 80 bits. The same operation is performed sequentially on all 63 blocks of the 10-second speech, resulting in a total bitstream size of 63 × 80 = 5040 bits.

[0051] like Figure 4 As shown, decoding and reconstruction (receiving end): The receiving end receives an 80-bit bitstream for each block. For the first block, decapsulation yields a 4×4 index matrix. Based on each index, the corresponding 128-dimensional codeword is retrieved from codebook E and reassembled into a quantized latent representation. .Will The input includes a 2D convolutional decoder and a decoder from the TSPN module; the output is a reconstructed Barker frequency cepstral coefficient feature block. .Should The input TSPN is then processed through three layers of one-dimensional convolutions (with kernel sizes of 5, 5, and 3) for residual refinement, resulting in the output of the reconstructed BFCC parameter sequence. Then, inverse normalization is performed using the same mean and standard deviation as the encoding end: The final reconstructed BFCC parameter sequence is obtained. The 63 reconstructed blocks are spliced ​​together in chronological order to obtain a complete 10-second BFCC parameter sequence (1000 frames in total, 17 dimensions per frame).

[0052] Using the above scheme, under the same 10-second Chinese speech input, the BFCC parameters were compressed and reconstructed using both the method of this invention and the traditional three-level vector quantization method. The root mean square error (RMSE) between the reconstructed BFCC and the original BFCC was calculated, and the results are as follows. Figure 5 As shown, the root mean square error (RMSE) of the present invention is reduced by more than 50%, and the inter-frame popping sound of the audio decoded using the reconstructed BFCC parameters is significantly reduced.

[0053] This invention also provides an ultra-low bit rate audio communication system based on a vector quantization variational autoencoder. The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method described above.

[0054] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned ultra-low bit rate audio communication method based on vector quantization variational autoencoder. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0055] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0056] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0057] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0058] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for ultra-low bit rate audio communication based on vector quantization variational autoencoder, characterized in that, The method includes encoding and decoding processes, wherein the encoding process includes: Extract the Barker frequency cepstral coefficient features from each frame of speech data, and construct a two-dimensional time-frequency feature block from the Barker frequency cepstral coefficient features of multiple consecutive frames of speech data. The two-dimensional time-frequency feature block is input into a preset two-dimensional convolutional encoder, which outputs a continuous latent representation. The two-dimensional convolutional encoder includes multiple convolutional layers. The continuous latent representation is input into a preset TAEM module, and an enhanced latent representation is obtained through the TAEM module. The TAEM module adopts a self-attention mechanism for global context enhancement. The output bitstream is constructed based on the enhanced latent representation and the preset codebook; The decoding process includes the following steps: The output bitstream is obtained, a quantized latent representation is obtained based on the output bitstream, and the quantized latent representation is input into a two-dimensional convolutional decoder to obtain a Barker frequency cepstral coefficient feature block. The Buck frequency cepstral coefficient feature block is input into a preset TSPN module, which includes multiple one-dimensional convolutional layers. The TSPN module is used to perform inter-frame smoothing optimization on the reconstructed parameters in a residual manner to obtain the reconstructed Buck frequency cepstral coefficient parameter sequence.

2. The ultra-low bit rate audio communication method based on vector quantization variational autoencoder according to claim 1, characterized in that, The decoding process further includes: The reconstructed Buck frequency cepstral coefficient parameter sequence is input into a preset neural vocoder synthesis network to obtain the final reconstructed speech signal.

3. The ultra-low bit rate audio communication method based on vector quantization variational autoencoder according to claim 1, characterized in that, In the step of inputting the two-dimensional time-frequency feature block into the preset two-dimensional convolutional encoder, the two-dimensional time-frequency feature block is normalized, and the normalized two-dimensional time-frequency feature block is input into the preset two-dimensional convolutional encoder.

4. The ultra-low bit rate audio communication method based on vector quantization variational autoencoder according to claim 1, characterized in that, In the step of inputting the two-dimensional time-frequency feature block into a preset two-dimensional convolutional encoder, the two-dimensional convolutional encoder includes a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, a third two-dimensional convolutional layer, multiple two-dimensional residual blocks, and a fourth two-dimensional convolutional layer arranged in sequence, and each two-dimensional residual block is provided with two convolutional layers.

5. The ultra-low bit rate audio communication method based on vector quantization variational autoencoder according to any one of claims 1 to 4, characterized in that, In the step of inputting the continuous latent representation into a preset TAEM module to obtain an enhanced latent representation: The two-dimensional time-frequency space of the continuous latent representation is flattened into a feature sequence of a preset length. Each position of the feature sequence corresponds to a first latent vector of a preset number of dimensions. Then, the scaled dot product self-attention is calculated to obtain the attention output. The attention output is residually concatenated with a flattened feature sequence of a preset length to obtain the enhanced latent representation.

6. The ultra-low bit rate audio communication method based on vector quantization variational autoencoder according to claim 1, characterized in that, In the step of constructing the output bitstream based on the enhanced latent representation and the preset codebook: For the second latent vector at each position in the enhanced latent representation, calculate the Euclidean distance with each codeword, and take the index of the codeword with the smallest Euclidean distance as the index corresponding to the second latent vector; Combine all the indices corresponding to the second potential vectors and encapsulate them into an output bitstream.

7. The ultra-low bit rate audio communication method based on vector quantization variational autoencoder according to claim 1, characterized in that, In the steps of acquiring the output bitstream, obtaining a quantized latent representation based on the output bitstream, and inputting the quantized latent representation into a two-dimensional convolutional decoder to obtain a Barker frequency cepstral coefficient feature block, the corresponding codebook determines the corresponding second latent vector based on the index in the output bitstream to obtain a quantized latent representation, and inputs it into the two-dimensional convolutional decoder. The two-dimensional convolutional decoder includes a first two-dimensional decoding convolutional layer, multiple two-dimensional residual blocks, a first two-dimensional transposed convolutional layer, and a second two-dimensional transposed convolutional layer arranged in sequence to obtain the Barker frequency cepstral coefficient feature block.

8. The ultra-low bit rate audio communication method based on vector quantization variational autoencoder according to claim 5, characterized in that, In the step of inputting the Buck frequency cepstral coefficient feature block into a preset TSPN module, the TSPN module includes multiple one-dimensional convolutional layers. The TSPN module is used to perform inter-frame smoothing optimization on the reconstructed parameters in a residual manner to obtain the reconstructed Buck frequency cepstral coefficient parameter sequence. The TSPN module includes three one-dimensional convolutional layers arranged in sequence, and the kernel sizes of the three one-dimensional convolutional layers are 5, 5 and 3 in sequence.

9. The ultra-low bit rate audio communication method based on vector quantization variational autoencoder according to claim 2, characterized in that, In the step of inputting the reconstructed Barker frequency cepstral coefficient parameter sequence into a preset neural vocoder synthesis network to obtain the final reconstructed speech signal, the reconstructed Barker frequency cepstral coefficient parameter sequence is combined with decoding parameters and input into the neural vocoder synthesis network. The decoding parameters include pitch and gain. The neural vocoder synthesis network adopts an LPCNet-like structure or a convolutional recursive hybrid structure.

10. An ultra-low bit rate audio communication system based on a vector quantization variational autoencoder, characterized in that, The system includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor executes the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps of the method as described in any one of claims 1 to 9.

Images