Speech reconstruction method and system based on entropy coding residual quantization and spectral repair

Abstract

The present disclosure provides a speech reconstruction method and system based on entropy coding residual quantization and spectrum repair, relating to the technical field of artificial intelligence speech signal processing, comprising: obtaining an original speech waveform, inputting the speech waveform into a neural speech coding and decoding model, first entering the encoder to map the input speech waveform into an acoustic latent representation, and then performing residual quantization layer by layer through a residual vector quantization module, introducing a dynamic layer number selection mechanism based on gating and an entropy regularization constraint, so that the bits are adaptively allocated between different speech segments, reconstructing the reconstructed acoustic features of the latent representation, inputting the reconstructed acoustic features into the decoder to restore the time domain waveform, and then mapping to the log amplitude spectrum domain through the spectrum repair module, predicting the residual in the log amplitude spectrum domain and performing confidence gating fusion to obtain a complex spectrum, and then outputting the reconstructed speech after time domain synthesis. The present disclosure can balance the high fidelity, intelligibility and reliability of transmission of speech at very low code rate.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence speech signal processing technology, specifically to a speech reconstruction method and system based on entropy coding residual quantization and spectrum restoration. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Low bit-rate speech coding technology has significant application value in modern communication systems, especially in special environments such as satellite communication, shortwave communication, underwater acoustic communication, and secure communication. Because these scenarios generally face problems such as limited bandwidth, poor channel quality, and complex environmental noise, achieving high-quality speech transmission at extremely low bit rates has become a core research problem in the field of speech signal processing. Taking satellite communication as an example, speech signals need to cross the atmosphere and space environment, resulting in extremely limited transmission bandwidth. In military or emergency communications, equipment often needs to operate under low power consumption and high interference conditions, requiring speech signals to be clear and intelligible while consuming as few resources as possible. Therefore, low bit-rate speech coding can not only improve communication efficiency but also play a crucial role in secure transmission and energy-saving applications.

[0004] With the development of deep learning technology, neural vocoders have gradually become a research hotspot in the field of speech encoding and decoding. A typical neural vocoder uses a convolutional neural network to downsample the input speech at the encoding end, designs a multi-level vector quantizer at the quantization end to map continuous features to discrete codeword indices, and uses a decoder at the decoding end to upsample and reconstruct the speech to generate time-domain speech. A discriminator is introduced in the training phase to enhance the generator's realism through adversarial training, thereby improving the naturalness and clarity of the synthesized speech.

[0005] However, existing neural vocoders still have the following problems:

[0006] (1) Speech signals themselves have multi-scale features, and different time scales and frequency ranges contain rich speech details. However, existing models are not good at capturing and fusing multi-scale features, which can easily cause information loss or redundancy, thus affecting the quality of synthesized speech.

[0007] (2) During upsampling and downsampling, high-frequency details often cannot be effectively recovered, resulting in blurry and distorted speech after decoding. Especially under low bit rate conditions, excessive compression of details significantly reduces the naturalness of the speech.

[0008] (3) During the quantization stage, the problem of uneven codebook utilization is prominent. Some codewords are idle for a long time, while others are used too frequently, resulting in a decline in expressive ability and quantization distortion.

[0009] (4) Insufficient modeling of phase information in the spectrum often leads to perceptual defects in synthesized speech, such as metallic sounds or transient distortion;

[0010] (5) From the perspective of bitstream organization, most current methods lack efficient side information compression and robust design. Once packet loss or asynchrony occurs, the decoding end often has difficulty recovering quickly, affecting the stability and reliability of actual communication. Summary of the Invention

[0011] To address the aforementioned issues, this disclosure proposes a speech reconstruction method and system based on entropy-coded residual quantization and spectrum restoration. It achieves adaptive bit allocation by introducing a dynamically gated residual vector quantization mechanism at the encoding end; at the decoding end, spectrum restoration technology is used to predict residuals in the logarithmic amplitude spectral domain and perform confidence-gated fusion, combined with phase gradient restoration to compensate for high-frequency loss; and at the bitstream organization level, an anchor point mechanism and a layer mask context entropy coding scheme are set up to enable random access and packet loss backoff, improving the system's robustness under complex network conditions.

[0012] According to some embodiments, the present disclosure adopts the following technical solutions:

[0013] Speech reconstruction methods based on entropy-coded residual quantization and spectral restoration include:

[0014] Acquire the raw speech waveform and perform preprocessing;

[0015] The preprocessed speech waveform is input into the neural speech encoding and decoding model, and the reconstructed speech is output.

[0016] The processing of the preprocessed speech waveform in the neural speech encoding / decoding model includes:

[0017] The input speech waveform first enters the encoder, which employs a multi-layer convolutional residual structure and hierarchical downsampling to map the input speech waveform into an acoustic latent representation. The acoustic latent representation is then input to the residual vector quantization module, which performs residual quantization layer by layer on the acoustic latent representation. A gated dynamic layer selection mechanism and entropy regularization constraint are introduced to adaptively allocate bits among different speech segments, resulting in quantized features. The quantized features are then reconstructed to obtain the reconstructed acoustic features of the latent representation. The reconstructed acoustic features are input to the decoder, which is then restored to a time-domain waveform through hierarchical upsampling and residual blocks. The time-domain waveform is then mapped to the logarithmic amplitude spectrum domain through a spectrum restoration module. The residual is predicted in the logarithmic amplitude spectrum domain and confidence-gated fusion is performed to obtain a complex spectrum. Finally, after time-domain synthesis, the reconstructed speech is output.

[0018] According to some embodiments, the present disclosure adopts the following technical solutions:

[0019] A speech reconstruction system based on entropy-coded residual quantization and spectrum restoration includes:

[0020] The data acquisition module is used to acquire the raw speech waveform and perform preprocessing.

[0021] The speech reconstruction module is used to input the preprocessed speech waveform into the neural speech encoding and decoding model and output the reconstructed speech.

[0022] The processing of the preprocessed speech waveform in the neural speech encoding / decoding model includes:

[0023] The input speech waveform first enters the encoder, which employs a multi-layer convolutional residual structure and hierarchical downsampling to map the input speech waveform into an acoustic latent representation. The acoustic latent representation is then input to the residual vector quantization module, which performs residual quantization layer by layer on the acoustic latent representation. A gated dynamic layer selection mechanism and entropy regularization constraint are introduced to adaptively allocate bits among different speech segments, resulting in quantized features. The quantized features are then reconstructed to obtain the reconstructed acoustic features of the latent representation. The reconstructed acoustic features are input to the decoder, which is then restored to a time-domain waveform through hierarchical upsampling and residual blocks. The time-domain waveform is then mapped to the logarithmic amplitude spectrum domain through a spectrum restoration module. The residual is predicted in the logarithmic amplitude spectrum domain and confidence-gated fusion is performed to obtain a complex spectrum. Finally, after time-domain synthesis, the reconstructed speech is output.

[0024] According to some embodiments, the present disclosure adopts the following technical solutions:

[0025] A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the speech reconstruction method based on entropy-coded residual quantization and spectral repair.

[0026] According to some embodiments, the present disclosure adopts the following technical solutions:

[0027] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the speech reconstruction method based on entropy coding residual quantization and spectrum repair.

[0028] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0029] This disclosed speech reconstruction method based on entropy coding residual quantization and spectrum restoration introduces a dynamic-gated residual vector quantization mechanism at the encoding end to achieve adaptive bit allocation, thereby improving compression efficiency and codebook utilization under limited bit rates. At the decoding end, a spectrum restoration mechanism is used to predict residuals in the logarithmic amplitude spectral domain and perform confidence-gated fusion, combined with phase gradient restoration to compensate for high-frequency missing values. At the same time, an anchor point mechanism and a layer mask context entropy coding scheme are set at the bitstream organization level to achieve random access and packet loss backoff, thereby improving the robustness of the system under complex network conditions.

[0030] This disclosed speech reconstruction method based on entropy-coded residual quantization and spectrum repair introduces a gated dynamic layer selection and entropy regularization constraint in the residual vector quantization stage, enabling bit allocation to adaptively adapt to the complexity of the speech content: for difficult segments such as plosives, fricatives, and rapid transients, the system automatically increases the effective layer depth to ensure detail; for silences and stable segments, the number of layers is reduced to save bits. This mechanism significantly alleviates the "over-matching / under-matching" contradiction caused by a fixed number of layers, while improving the balance and effective capacity of codeword usage, reducing the accumulation and diffusion of quantization distortion over time, and making the reconstructed speech more stable in terms of clarity and intelligibility. It can maintain high speech quality under extremely low bitrate conditions and is suitable for scenarios such as low-bandwidth voice communication, satellite and shortwave transmission, IoT edge nodes, and voice archive compression storage.

[0031] This disclosed speech reconstruction method based on entropy-coded residual quantization and spectrum restoration utilizes a spectrum restoration mechanism at the decoding end that centers on residual refinement in the logarithmic amplitude spectrum domain, supplemented by confidence gating and an optional phase correction path. This mechanism can specifically compensate for the loss of high-frequency details and artifact accumulation caused by quantization. Compared with schemes that rely solely on time-domain or single amplitude spectrum constraints, this module exhibits a more natural energy distribution at harmonic structures, hissing boundaries, and transient zero crossings, significantly reducing perceptual defects such as "metallic" and "gritty" sounds. Simultaneously, it employs careful gain control in distorted regions to avoid pumping and ringing caused by over-restoration, thereby achieving higher subjective naturalness and spatiality.

[0032] This disclosed speech reconstruction method based on entropy-coded residual quantization and spectrum repair significantly reduces side information overhead at the bitstream organization level by performing contextual probability modeling on the layer mask and employing arithmetic coding compression. Simultaneously, it introduces anchor packets inserted at fixed intervals to record quantization state and phase reference, enabling the decoder to have random start-up and rapid packet loss backoff capabilities. This design significantly improves continuous playback and interactive experience under complex network conditions without significantly increasing the overall bitrate, reducing stuttering, audio interruptions, and the duration of significant distortion, thus improving the stability and availability of end-to-end services.

[0033] This disclosed speech reconstruction method based on entropy-encoded residual quantization and spectral inpainting, along with a multi-domain, multi-scale discriminator and mutual exclusion regularization during the training phase, enables the system to simultaneously constrain generation quality from three dimensions: periodic structure, temporal scale, and spectral texture, avoiding training efficiency losses caused by overlapping discriminator concerns. Combined with multi-resolution STFT and feature matching loss, the model maintains high harmonic consistency, detail resolution, and speech flow coherence even at low bit rates, resulting in a systematic improvement in both subjective and objective metrics.

[0034] The speech reconstruction method based on entropy-coded residual quantization and spectrum repair disclosed herein also has advantages in engineering and deployment. Dynamic layer selection and context compression reduce the average bit rate and computational burden, the anchor point system facilitates synchronous maintenance and version evolution during long-term operation, and the spectrum repair mechanism adopts a lightweight U-Net structure, which is easy to use in conjunction with inference optimizations such as quantization, distillation, and pruning, thereby achieving low-latency and low-power operation on mobile and edge devices. In summary, this disclosure can balance speech quality, robustness, and transmission efficiency in ultra-low bit rate scenarios such as 0.6-0.8kbps, and has significant application and industrialization value. Attached Figure Description

[0035] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0036] Figure 1 This is a schematic diagram of the structure of the neural speech encoding and decoding model according to an embodiment of the present disclosure;

[0037] Figure 2 This is a flowchart of entropy-encoded dynamic residual quantization and spectrum restoration according to an embodiment of this disclosure;

[0038] Figure 3 This is a flowchart illustrating the spectrum repair mechanism of an embodiment of this disclosure;

[0039] Figure 4 This is a schematic diagram of the entropy-encoded dynamic residual quantization module in an embodiment of this disclosure. Detailed Implementation

[0040] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0041] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0042] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0043] Example 1

[0044] One embodiment of this disclosure provides a speech reconstruction method based on entropy-coded residual quantization and spectral restoration, the method steps of which include:

[0045] Step 1: Obtain the raw speech waveform and perform preprocessing;

[0046] Step 2: Input the preprocessed speech waveform into the neural speech encoding and decoding model, and output the reconstructed speech;

[0047] The processing of the preprocessed speech waveform in the neural speech encoding / decoding model includes:

[0048] The input speech waveform first enters the encoder, which employs a multi-layer convolutional residual structure and hierarchical downsampling to map the input speech waveform into an acoustic latent representation. The acoustic latent representation is then input to the residual vector quantization module, which performs residual quantization layer by layer on the acoustic latent representation. A gated dynamic layer selection mechanism and entropy regularization constraint are introduced to adaptively allocate bits among different speech segments, resulting in quantized features. The quantized features are then reconstructed to obtain the reconstructed acoustic features of the latent representation. The reconstructed acoustic features are input to the decoder, which is then restored to a time-domain waveform through hierarchical upsampling and residual blocks. The time-domain waveform is then mapped to the logarithmic amplitude spectrum domain through a spectrum restoration module. The residual is predicted in the logarithmic amplitude spectrum domain and confidence-gated fusion is performed to obtain a complex spectrum. Finally, after time-domain synthesis, the reconstructed speech is output.

[0049] As one embodiment, this disclosure presents a speech reconstruction method based on entropy coding residual quantization and spectrum repair. This method introduces a residual vector quantization mechanism based on dynamic gating at the encoding end to achieve adaptive bit allocation, thereby improving compression efficiency and codebook utilization under limited bit rates. At the decoding end, a spectrum repair mechanism is used to predict residuals in the logarithmic amplitude spectral domain and perform confidence-gated fusion, combined with phase gradient repair to compensate for high-frequency missing values. At the same time, an anchor point mechanism and a layer mask context entropy coding scheme are set at the bitstream organization level to achieve random access and packet loss backoff, thereby improving the robustness of the system under complex network conditions.

[0050] This disclosure presents a neural speech encoding and decoding model, which includes an encoder, a residual vector quantization module, a bitstream organization module, a decoder, and a spectrum restoration module. The encoder and decoder adopt a mirrored SEANet structure, and the channel base width is denoted as . The downsampling ratios are set to [10, 8, 4, 1]. The residual vector quantization module mainly consists of a residual vector quantizer (RVQ), which includes... Layer codebook, number The size of the codebook is The quantization dimension is aligned with the latent dimension. Multi-scale STFT constraints use several sets ( )combination, The number of points representing the Fourier transform (i.e., the window length). Representing the frame shift length, the model can learn the spectral characteristics of speech at different scales by using multiple sets of parameters, typically (512, 80), (1024, 160), and (256, 40). The Spectrum Restoration Module (SRM) adopts a lightweight U-Net architecture with 4 layers each for encoding and decoding, and a channel width of 16-128. Its output includes the logarithmic magnitude spectrum residual. With confidence plot In the extended configuration, it simultaneously outputs the phase gradient correction amount. The discriminator employs a multi-period (MPD) and multi-scale (MSD) structure, combined with a multi-scale STFT discriminator (MS-STFTD) to form a multi-domain constraint. A layer mask context model and an arithmetic encoder are set on the bitstream side, with fixed intervals... Frame insertion anchor packet (GOP anchor). 50-200 frames can be used to balance latency and redundancy. The specific implementation process of the speech reconstruction method based on entropy coding residual quantization and spectrum repair disclosed in this paper is as follows:

[0051] Step 1: Obtain the raw speech waveform and perform preprocessing;

[0052] The original speech sampling rate was set to 8kHz, and the frame shift and segment length were set for mini-batch training and streaming inference, with reading and writing typically performed in 20ms granularity.

[0053] Furthermore, the preprocessing steps include: normalizing or peak limiting the input raw speech waveform, removing the DC component, and performing light pre-noise reduction as needed.

[0054] During the training phase, long speech segments are divided into fixed-length segments (e.g., 1.0s-1.5s) and randomly pruned and perturbed with random gain. To improve robustness, room reverberation, band-limited filtering, and background noise can be added with a certain probability. During the inference phase, audio blocks are collected in streaming mode using a circular buffer. Each time a basic frame shift is collected, an encoding-quantization-packing process is triggered.

[0055] Step 2: Input the preprocessed speech waveform into the neural speech encoding / decoding model, and output the reconstructed speech. The neural speech encoding / decoding model includes an encoder, a residual vector quantization module, a code stream organization module, a decoder, and a spectrum restoration module. The specific processing steps include:

[0056] Step 21: First, the encoder employs a multi-layer convolutional residual structure and hierarchical downsampling to map the input speech waveform into an acoustic latent representation, including:

[0057] The speech waveform input to the neural speech encoding / decoding model first enters the encoder, which adopts the SEANet structure and consists of several residual convolutional units cascaded with hierarchical downsampling modules. Specifically, these residual convolutional units are residual blocks that include channel attention mechanisms, and internally contain pointwise convolution, dilated convolution, and normalization / activation order; skip connections are maintained between layers to facilitate gradient and information flow. Furthermore, modules such as Long Short-Term Memory (LSTM) networks can be integrated into the model to enhance the ability to model time series. The speech signal is then sampled by the hierarchical downsampling module to obtain a time series with a set length. Potential tensors To facilitate the operation of the residual vector quantizer (RVQ), the tensor dimensions are uniformly organized to obtain the acoustic latent representation. ,in B This refers to the batch size. D The number of channels for the potential representation. T' This represents the length of the time frame after downsampling.

[0058] Step 22: Input the acoustic latent representation into the residual vector quantization module, perform residual quantization on the acoustic latent representation layer by layer, introduce a gate-based dynamic layer selection mechanism and entropy regularization constraint to adaptively allocate bits among different speech segments, including:

[0059] The residual vector quantization module is a residual vector quantizer (RVQ), which includes... Layer codebook, number The size of the codebook is The quantified dimension is aligned with the potential dimension.

[0060] Residual Vector Quantizer (RVQ) for Acoustic Latent Characterization Perform residual quantization layer by layer, the first Layered codebook Nearest neighbor quantization current residual Get the index With reconstructed vector and update the residuals. .

[0061] This disclosure introduces a dynamic gating mechanism based on Gumbel-Softmax (a gating-based dynamic layer selection mechanism) to adaptively select the number of layers under a given bitrate: a lightweight scoring network is used to generate an importance score for each layer. The gating probability is obtained by Gumbel-Softmax normalization. Average pooling is performed on the time dimension to obtain the segment-level score. and with the target bitrate The mapping relationship between them can be calculated using layer depth. The implementation method is based on the average bit budget per layer. The bit rate consumption of the approximation layer ( (for frames per second), searching for the satisfying The smallest ,in Amortize bits for side information (layer mask, anchor points, etc.).

[0062] During training, a straight-through estimator (STE) is used to calculate the gradient of discrete choices; during inference, scores are calculated at the fragment level. Select from high to low Layer as an activation layer set Unselected layers do not generate codewords and only participate in regularization and distillation loss during training. To balance codeword usage, an empirical distribution is applied to each codebook. Adding normalized entropy regularization It is also assigned a small weight to avoid disturbing the main objective.

[0063] Step 23: The bitstream organization module encapsulates the quantization index and layer mask, uses the context model for probabilistic modeling, and employs arithmetic coding compression to reduce information overhead;

[0064] Specifically, the stream organization module will activate the layer set. Encoded as a length of binary layer mask ,in Indicates the first The layer is activated. The layer mask exhibits Markov property over time, using a first-order context model. The conditional distribution is estimated and written into the bitstream using arithmetic coding, thereby constraining the side information bits to a minimal overhead range. Subsequently, the codeword index sequences of each layer are written in layer order. Quantitative features are obtained;

[0065] To improve error recovery capabilities, every The frame is written to the anchor packet, which contains an RVQ residual state summary, necessary normalized statistics, and phase reference hash. The bitstream header records metadata such as sampling rate, encoder version, STFT group, and discriminator signature, facilitating compatibility between different model versions at the decoding end.

[0066] Step 24: The decoder reads the quantization features and first parses the header information and anchor points; if it is a random start-up or packet loss recovery scenario, it recovers the RVQ state starting from the most recent anchor point, and then decodes the layer mask in chronological order. By combining the codeword indexes of each layer, the quantized features are reconstructed and then stacked layer by layer to obtain the reconstructed acoustic features of the latent representation. and organize it into The layout prioritizes channels that align with the decoder input.

[0067] Step 25: Reconstruct the acoustic features The input to the SEANet-structured decoder is then restored to a time-domain waveform through hierarchical upsampling and residual blocks. To reduce boundary effects, an overlap-addition (OLA) strategy is used to stitch consecutive frames into a complete stream; in streaming mode, sliding window alignment is used to ensure phase continuity, and an extremely lightweight limiter is used to suppress occasional spikes.

[0068] Step 26: Map the time-domain waveform to the logarithmic amplitude spectrum domain through the spectrum repair module, predict the residual in the logarithmic amplitude spectrum domain and perform confidence-gated fusion to obtain the complex spectrum;

[0069] Specifically, the time-domain waveform The complex spectrum is obtained by projecting multiple sets of short-time Fourier transforms (STFTs) onto the frequency domain. Take the logarithm of the amplitude component and add a numerical stability term. ,get As input to the Spectrum Repair Module (SRM), this input is processed through a lightweight U-Net structure. This U-Net network includes a downsampling encoding path composed of multiple convolutional, normalization, and ReLU activation functions, and an upsampling decoding path containing skip connections, thus effectively fusing multi-scale spectral features. The residual is obtained after processing by the U-Net structure. With confidence plot .

[0070] The restored logarithmic amplitude is obtained by gating. Then the exponent is restored to With phase branch enabled, the phase gradient correction is predicted simultaneously. , forming phase The final complex spectrum is The reconstructed speech waveform is obtained by performing inverse short-time Fourier transform (ISTFT) with the same window size and overlap rate as the encoder in the time domain. The output of the phase branch is usually... Amplitude is limited and constrained by a scaling factor to avoid introducing unstable phase jumps; confidence plots are used to suppress over-repair of SRM in uncertain regions and reduce "pumping sound" and "metallic sound".

[0071] As one embodiment, the speech reconstruction method based on entropy-coded residual quantization and spectral repair disclosed herein concatenates the entire process described in steps 21-26 into an end-to-end graph during the training phase. It jointly utilizes waveform L1, multi-resolution STFT, feature matching, and adversarial loss, and employs a multi-period discriminator, a multi-scale discriminator, and a multi-scale STFT discriminator for multi-domain optimization. Specifically:

[0072] Reconstruction loss is performed in the waveform domain. Combined with multi-resolution STFT distance, the spectral distance is selected as a logarithmic amplitude and phase consistency metric, and the original speech is... x Reconstructing speech G ( z ), Reconstruction losses:

[0073] L_recon = || x - G ( z )||_1 + α * L_stft( x , G ( z ))

[0074] The adversarial loss adopts a logistic form. The discriminators include three types: MPD, MSD, and MS-STFTD, which respectively constrain periodic structure, time scale, and spectral structure. The logistic form of the adversarial loss is as follows: G For generator, D For the discriminator:

[0075] L_adv( G , D ) = E_x[log D( x )] + E_z[log(1 - D( G ( z )))]

[0076] Feature matching loss is introduced to stabilize training and improve perceptual quality. This loss is achieved by minimizing the L1 distance of the feature maps in the intermediate layers of the discriminator network, as shown in the following formula:

[0077] L_fm(G , D ) = E _{ x , z}[Σ_{ i =1 to L} (1 / N _ i ) * || D _ i ( x ) - D _ i ( G ( z ))||_1]

[0078] Mutual exclusion regularization or mutual information penalty is added between discriminators to encourage them to learn complementary features rather than simple repetition. This is achieved by minimizing the cosine similarity between features extracted by different discriminators (such as MPD and MSD).

[0079] L_mutex = |cos(f_mpd( x ), f_msd( x ))|

[0080] Where f_mpd and f_msd represent the feature vectors extracted from MPD and MSD, respectively.

[0081] On the RVQ side, codeword entropy regularization is used to suppress the "hot codeword" phenomenon, and sparsification or temperature annealing is applied to dynamic gating to stabilize selection. In terms of training strategy, the high-resolution discriminator and phase branch are initially disabled, convergence is achieved first with amplitude and low-resolution spectral terms, and then gradually enabled; fragment length and batch size are linearly increased according to GPU memory; the optimizer uses Adam / AdamW, with an initial learning rate... The magnitude can be adjusted and combined with cosine annealing or multi-stage attenuation. To better suit real-world scenarios, noise / reverberation / band-limited distortion can be injected with a 10–20% probability; for edge deployment, distillation, INT8 quantization, and structured pruning can be used as supplementary techniques.

[0082] Furthermore, for scenarios requiring even greater bit squeezing, the gating score is time-varying by combining Voice Activity Detection (VAD) with transient detectors (such as plosive / friction detection). This involves appropriately increasing the upper limit for spoken segments or highly complex transient segments, and decreasing it for silent or steady vowels. Furthermore, the layer mask sequence on the time axis is used as a Markov chain for context entropy encoding. This mechanism significantly reduces the temporal mean of side information and total bit rate while maintaining quality in critical segments.

[0083] The decoder continuously monitors the continuity of the layer mask and codeword stream. When a loss is detected or a CRC check fails, it immediately backtracks to the nearest anchor point, reconstructs the RVQ state, and continues decoding from there. If the loss occurs within the anchor interval and the probability of recoverable context is high, a soft-filling strategy can be used to perform maximum a posteriori (MAP) estimation on the missing mask and attempt a smooth transition; if this fails, a forced backtracking is performed to ensure overall audio quality consistency.

[0084] To ensure compatibility during long-term evolution, the bitstream header records the model version, STFT group, RVQ architecture summary, and SRM configuration signature. The decoding end can establish a backward compatibility table, ensuring stable decoding of older bitstream versions even when SRM is disabled or phase branching is disabled. For end-devices, the target bitrate, upper limit of the number of layers, and whether phase branching is enabled can be agreed upon during session establishment through capability negotiation.

[0085] Simulation Experiment

[0086] This disclosure uses the publicly available librisspeech corpus, with a sampling rate of 16kHz, which is then downsampled to 8kHz. Training data consists of 800 hours of speech from the librisspeech training set, and testing data consists of 10 hours of speech from the librisspeech test set. To further evaluate the model's generalization ability, additional speech tests were conducted on the LJSpeech and THCHS-30 datasets.

[0087] The objective metrics used in the experiment were PESQ (Perceptual Speech Quality Assessment), STOI (Short-Time Objective Intelligibility), and LSD (Log-Spectral Distance).

[0088] MUSHRA is a commonly used subjective scoring method in speech enhancement and speech quality assessment. Subjects simultaneously hear multiple audio samples, including the test sample, a hidden reference, and anchors, and directly compare their scores. The average score of all subjects' scores is then calculated to obtain the corresponding MUSHRA value. A hidden reference is an unprocessed raw reference signal (highest quality) included in the test, but its identity is not revealed to the subject. This is used to verify the objectivity of the scoring; ideally, the reference should receive the highest score. Anchors are deliberately added low-quality audio (such as low-bitrate encoded or severely distorted samples) as a "bottom line" for the scoring benchmark, helping to calibrate the subject's scoring scale. The MUSHRA score range is 0-100.

[0089] To assess subjective quality, this publication uses the MUSHRA index: 10 listeners (5 women and 5 men, aged 23–32) rate 20 randomly selected utterances from the LibriSpeech test set.

[0090] Experimental results show that this invention significantly outperforms existing methods in both objective metrics and subjective evaluation at ultra-low bit rates of 0.6–0.8 kbps, demonstrating significant application potential.

[0091] Example 2

[0092] One embodiment of this disclosure provides a speech reconstruction system based on entropy-coded residual quantization and spectral repair, comprising:

[0093] The data acquisition module is used to acquire the raw speech waveform and perform preprocessing.

[0094] The speech reconstruction module is used to input the preprocessed speech waveform into the neural speech encoding and decoding model and output the reconstructed speech.

[0095] The processing of the preprocessed speech waveform in the neural speech encoding / decoding model includes:

[0096] The input speech waveform first enters the encoder, which employs a multi-layer convolutional residual structure and hierarchical downsampling to map the input speech waveform into an acoustic latent representation. The acoustic latent representation is then input to the residual vector quantization module, which performs residual quantization layer by layer on the acoustic latent representation. A gated dynamic layer selection mechanism and entropy regularization constraint are introduced to adaptively allocate bits among different speech segments, resulting in quantized features. The quantized features are then reconstructed to obtain the reconstructed acoustic features of the latent representation. The reconstructed acoustic features are input to the decoder, which is then restored to a time-domain waveform through hierarchical upsampling and residual blocks. The time-domain waveform is then mapped to the logarithmic amplitude spectrum domain through a spectrum restoration module. The residual is predicted in the logarithmic amplitude spectrum domain and confidence-gated fusion is performed to obtain a complex spectrum. Finally, after time-domain synthesis, the reconstructed speech is output.

[0097] Example 3

[0098] One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions. When these computer instructions are executed by a processor, they implement the speech reconstruction method based on entropy-coded residual quantization and spectrum repair.

[0099] Example 4

[0100] One embodiment of this disclosure provides an electronic device, including a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the speech reconstruction method based on entropy coding residual quantization and spectrum repair.

[0101] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0103] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. Speech reconstruction method based on entropy coding residual quantization and spectral repair, characterized in that, include: Acquire the raw speech waveform and perform preprocessing; The preprocessed speech waveform is input into the neural speech encoding and decoding model, and the reconstructed speech is output. The processing of the preprocessed speech waveform in the neural speech encoding / decoding model includes: The input speech waveform first enters the encoder, which uses a multi-layer convolutional residual structure and hierarchical downsampling to map the input speech waveform into an acoustic latent representation. The acoustic latent representation is then input into the residual vector quantization module, which performs residual quantization layer by layer on the acoustic latent representation. A gate-based dynamic layer selection mechanism and entropy regularization constraint are introduced to adaptively allocate bits among different speech segments to obtain quantized features. The quantized features are then reconstructed to obtain the reconstructed acoustic features of the latent representation. The reconstructed acoustic features are input into the decoder, which is then restored to a time-domain waveform through hierarchical upsampling and residual blocks. The time-domain waveform is then mapped to the logarithmic amplitude spectrum domain through a spectrum restoration module. The residual is predicted in the logarithmic amplitude spectrum domain and confidence-gated fusion is performed to obtain a complex spectrum. Finally, after time-domain synthesis, the reconstructed speech is output. The process involves inputting the acoustic latent representation into the residual vector quantization module, performing residual quantization layer by layer on the acoustic latent representation, and introducing a gate-based dynamic layer selection mechanism and entropy regularization constraints to adaptively allocate bits among different speech segments, including: Residual quantization is performed layer by layer on the acoustic latent representation. At each layer, the current residual is quantized with the nearest neighbor of the set codebook to obtain the index and the reconstruction vector, and the residual is updated. To adaptively select the number of layers at a given code rate, a dynamic layer selection mechanism based on gating is introduced. A lightweight scoring network is used to generate importance scores for each layer. The gating probabilities are obtained through dynamic gating normalization. The segment-level scores are obtained by average pooling in the time dimension. The available layer depth is calculated by the mapping relationship between the segment-level scores and the target code rate n . The top n layers are selected as the active layer set according to the segment-level scores from high to low. To balance codeword usage, normalized entropy regularization is added to the empirical distribution on each codebook, with small weights.

2. The speech reconstruction method based on entropy coding residual quantization and spectral repair as claimed in claim 1, wherein, The process of acquiring the original speech waveform and performing preprocessing includes: The original speech waveform is acquired at a set sampling rate. The acquired original speech waveform is then normalized or peak-limited, DC components are removed, and light pre-noise reduction is performed as a preprocessing operation.

3. The speech reconstruction method based on entropy coding residual quantization and spectral repair as claimed in claim 1, wherein, The input speech waveform first enters the encoder, which employs a multi-layer convolutional residual structure and hierarchical downsampling to map the input speech waveform into an acoustic latent representation, including: The encoder adopts the SEANet structure, which consists of several residual convolutional units cascaded with a hierarchical downsampling structure. The residual convolutional unit contains pointwise convolution, dilated convolution, and normalization / activation order. After multiple levels of downsampling, a latent tensor with a set time length is obtained. The latent tensor is then unified in dimension to obtain an acoustic latent representation.

4. The speech reconstruction method based on entropy coding residual quantization and spectral repair as claimed in claim 1, wherein, The activation layer set is encoded into a binary layer mask of length . The layer mask exhibits Markov property on the time axis. The conditional distribution is estimated using a first-order context model and written into the bit stream using arithmetic coding, thereby constraining the side information bits to a very small overhead range. Subsequently, the codeword index sequence of each layer is written in layer order to obtain the quantized features.

5. The speech reconstruction method based on entropy coding residual quantization and spectral repair as claimed in claim 1, wherein, The reconstructed acoustic features obtained by reconstructing the quantized features to obtain the potential representation include: Read the quantization features and first parse the header information and anchor points; if it is a random start-up or packet loss recovery scenario, start the recovery of RVQ state from the nearest anchor point, then solve the layer mask and codeword index of each layer in time order, reconstruct the quantization features and stack them layer by layer to obtain the reconstructed acoustic features of the potential representation, and arrange them into a channel priority layout that conforms to the decoder input.

6. The speech reconstruction method based on entropy coding residual quantization and spectral repair as claimed in claim 1, wherein, The time-domain waveform is mapped to a log-magnitude spectrum domain by a spectrum repair module, residual errors are predicted in the log-magnitude spectrum domain, and confidence gated fusion is performed to obtain a complex spectrum, including: The reconstructed acoustic feature of the latent representation is input into a decoder of the SEANet structure, restored to a time-domain waveform through hierarchical upsampling and a residual block, projected to a frequency domain through multiple groups of STFT, and the amplitude part is taken as a logarithm and added with a numerical stabilizing item, residual errors and a confidence map are obtained through a light U-Net, a repaired log-magnitude is obtained in a gated manner, a phase gradient correction amount is predicted, a phase is formed, and a final complex spectrum is obtained based on the phase.

7. Speech reconstruction system based on entropy coding residual quantization and spectral repair, implementing the speech reconstruction method based on entropy coding residual quantization and spectral repair according to any one of claims 1 to 6, characterized in that, Comprise: The data acquisition module is used for acquiring an original speech waveform and performing preprocessing; The speech reconstruction module is used for inputting the preprocessed speech waveform into a neural speech coding and decoding model to output a reconstructed speech. The processing process of the preprocessed speech waveform in the neural speech coding and decoding model comprises: The input speech waveform first enters an encoder, the encoder adopts a multi-layer convolution residual structure and hierarchical downsampling to map the input speech waveform to an acoustic latent representation, the acoustic latent representation is input into a residual vector quantization module, residual quantization is performed on the acoustic latent representation layer by layer, a dynamic layer number selection mechanism based on gating and an entropy regularization constraint are introduced to adaptively allocate bits between different speech segments, a quantized feature is obtained, the quantized feature is reconstructed to obtain a reconstructed acoustic feature of the latent representation, the reconstructed acoustic feature is input into a decoder, restored to a time-domain waveform through hierarchical upsampling and a residual block, mapped to a log-magnitude spectrum domain through a spectrum repair module, residual errors are predicted in the log-magnitude spectrum domain, and confidence gated fusion is performed to obtain a complex spectrum, and the reconstructed speech is output after time-domain synthesis.

8. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium is used for storing computer instructions, and the computer instructions are executed by the processor to implement the speech reconstruction method based on entropy coding residual quantization and spectrum repair according to any one of claims 1-6.

9. An electronic device, comprising: Comprise: A processor, a memory and a computer program; wherein the processor is connected with the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the speech reconstruction method based on entropy coding residual quantization and spectrum repair according to any one of claims 1-6.

Images