Neural vocoder training method and system based on residual modeling and time-frequency perception constraint

Abstract

The application provides a neural vocoder training method and system based on residual modeling and time-frequency perception constraint, comprising: receiving speech sample values and linear prediction coefficients, calculating linear prediction values and mapping to obtain real excitation categories, inputting speech frame features into a neural vocoder network to obtain a probability distribution, weighted sum to obtain a predicted excitation and add the linear prediction value to obtain a differentiable predicted waveform; based on the predicted waveform, cross-entropy loss, time-frequency perception loss based on synchronous compression transformation and low-order cepstrum envelope loss are jointly optimized to construct a total loss function; a gradual weight adjustment strategy is used to train the network, the weights of the time-frequency perception loss and the cepstrum envelope loss are linearly increased from zero to a preset target value, and after the training is completed, the network is deployed to a low-code-rate speech communication terminal. The application can significantly improve the spectral fidelity and subjective sound quality of low-code-rate neural vocoder synthesized speech without increasing the inference end code rate and computational complexity.

Description

Technical Field

[0001] This invention relates to the fields of speech coding, neural vocoders and speech synthesis technology, and in particular to a neural vocoder training method and system based on residual modeling and time-frequency perception constraints. Background Technology

[0002] In low-bit-rate voice communication scenarios, such as satellite communication, emergency communication, and narrowband IoT, bandwidth resources are extremely limited, typically requiring a target bit rate of 1 kbps or even lower for voice coding. To achieve intelligible and natural speech reconstruction under such extreme conditions, existing vocoder technologies generally employ a recursive generative architecture centered on linear predictive coding. This type of method first extracts low-bit-rate acoustic features such as Barker frequency cepstral coefficients (BFCC), pitch period, and pitch correlation from the speech frame. Then, it recovers the linear prediction coefficients and calculates the linear prediction value to approximate the current sampling point. Simultaneously, it uses a neural network to perform per-sample-point probability modeling of the linear prediction residual (i.e., the excitation signal), trains the network using cross-entropy loss to output the excitation category distribution, and finally adds the predicted excitation and the linear prediction value to reconstruct the time-domain waveform. This technical approach has become the mainstream solution in the field of low-bit-rate vocoders because it can maintain basic speech intelligibility at extremely low bit rates.

[0003] However, the aforementioned mainstream training frameworks exhibit significant audio quality bottlenecks under low bitrate quantization input conditions. The core supervisory signal of these methods is the cross-entropy loss of the excitation category per sample point. Their optimization objective is essentially to achieve accuracy in the residual sign and amplitude levels at each sample point. However, accuracy per sample point does not necessarily guarantee that the overall shape of the synthesized speech in the frequency domain is consistent with real speech. Small errors at multiple sample points can accumulate into significant spectral distortion in the frequency domain. This contradiction between time-domain accuracy and frequency-domain accuracy is further amplified under low bitrate conditions. The underlying reason is that after low bitrate quantization of acoustic features such as BFCC, the quantization error is not uniformly distributed across all frequency bands, but is significantly amplified in the formant band and the mid-to-high frequency transition region. This causes the quantized linear prediction coefficients to deviate from their true values, resulting in structural deviations in the spectral envelope of the synthesized speech. Simultaneously, the dynamic range compression of the residual signal caused by quantization makes the cross-entropy loss prone to gradient saturation in the later stages of training, reducing the magnitude of network parameter updates. The subtle structure of the spectral envelope cannot be fully optimized through the indirect path of residual prediction. The above factors together lead to common quality problems in existing methods under low bitrate input, such as inconsistency between the synthesized speech spectral envelope and real speech, high-frequency energy suppression in transition frames, and harmonic structure distortion. In the frequency domain, this manifests as formant shift or broadening, insufficient filling of spectral valleys, and attenuation of high-frequency harmonics. In the time domain, it manifests as inter-frame discontinuity of waveform envelope and blurred transient response. Ultimately, the synthesized speech exhibits obvious mechanical and artificial characteristics, with a decrease in objective loss value but extremely limited improvement in subjective sound quality.

[0004] To address this issue, some studies have attempted to introduce spectral loss functions into vocoder training to directly constrain the frequency domain characteristics of synthesized speech. For example, in the training of non-LPC generative vocoders, multi-resolution short-time Fourier transform loss has been widely adopted and proven to effectively improve synthesized sound quality. However, directly transferring this type of loss to recursive generative vocoders based on linear prediction residual modeling faces two inherent limitations. First, the conventional short-time Fourier transform is constrained by the Heisenberg uncertainty principle, resulting in a diffuse distribution of time-spectral energy in the time-frequency plane. For nonlinear distortion components generated under low bit-rate quantization input, their energy is diffused over a wide frequency band, leading to insufficient sensitivity of the loss function to such distortions and difficulty in providing effective gradient supervision. Second, the gradient directions of the spectral loss and the original cross-entropy loss may be inconsistent in the early stages of training. Simple superposition can lead to training oscillations or even divergence, and existing solutions lack effective adjustment mechanisms for the synergistic optimization of the two types of losses. In addition, the above attempts mostly focus on constraints at the time-spectrum level, and lack specific supervision over the long-term consistency of the global spectral envelope and formant structure of speech, resulting in a lack of naturalness in the timbre of the entire speech. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a neural vocoder training method and system based on residual modeling and time-frequency perception constraints, in order to solve the problems of distortion of the spectral envelope and formant structure of synthesized speech under low bit rate quantization conditions, attenuation of high-frequency harmonics in transition frames and discontinuity of time-frequency morphology, and insufficient sensitivity of conventional spectral loss to nonlinear distortion leading to training supervision failure in the prior art.

[0006] On one hand, the present invention provides a neural vocoder training method based on residual modeling and time-frequency sensing constraints, the method comprising: The system receives the current real speech sample value and the linear prediction coefficient of the current speech frame, calculates the linear prediction value based on the linear prediction coefficient and historical sample values, performs μ-law mapping on the difference between the real speech sample value and the linear prediction value to obtain the real excitation category, inputs the speech frame features of the current moment into the neural vocoder network to obtain the probability distribution of the current excitation category, performs weighted summation of the μ-law inverse mapping values ​​corresponding to each excitation category based on the probability distribution to obtain the predicted excitation, and adds the predicted excitation to the linear prediction value to obtain a differentiable predicted waveform. Based on the differentiable predicted waveform and the real waveform formed by the real speech sample values, three types of complementary constraints are jointly optimized. These three types of complementary constraints include: excitation prediction cross-entropy loss acting on each sample point, time-frequency perception loss based on synchronous compression transform, and low-order cepstral envelope loss. The time-frequency perception loss based on synchronous compression transform is calculated as follows: short-time Fourier transforms are performed on both the predicted waveform and the real waveform to obtain their respective short-time Fourier transform coefficients; instantaneous frequency estimation is calculated based on the short-time Fourier transform coefficients; and the time-frequency coefficients are synchronously compressed from their original positions to a new position determined by the instantaneous frequency estimation to obtain the synchronously compressed transform. Time-frequency spectrum; under multiple resolution configurations with different window lengths and frame shifts, the amplitude spectrum loss and logarithmic amplitude spectrum loss of the predicted waveform and the real waveform in the synchronous compression transform domain are calculated respectively, and the losses at each resolution are summed to obtain the multi-resolution time-frequency spectrum loss; additional weights are applied to the formant sensitive frequency band to obtain the time-frequency sensing loss based on synchronous compression transform; the low-order cepstral envelope loss is calculated as follows: Mel spectrum is extracted from the predicted waveform and the real waveform respectively, and after logarithmic operation, discrete cosine transform is performed to extract the low-order cepstral coefficients. The mean square error between the low-order cepstral coefficients of the predicted waveform and the real waveform is calculated to obtain the low-order cepstral envelope loss; The total loss function is constructed by weighted summing of the cross-entropy loss, the time-frequency sensing loss based on synchronous compression transform, and the low-order cepstral envelope loss. The neural vocoder network is trained using a progressive weight adjustment strategy: in the initial few rounds of training, the weights of the time-frequency sensing loss and the cepstral envelope loss are linearly increased from zero to a preset target value, while the weight of the cross-entropy loss remains constant. The network parameters are updated with the goal of minimizing the total loss function until training is complete. The trained neural vocoder network is then deployed to a low-bit-rate voice communication terminal. The neural vocoder network is used to reconstruct the received quantized acoustic features into a time-domain speech pulse-code modulation signal.

[0007] In some embodiments of the present invention, additional weights are applied to the resonant sensitive frequency band, including: The loss value belonging to the formant sensitive frequency band in the multi-resolution time-frequency loss is multiplied by a first weighting coefficient, and the loss value not belonging to the formant sensitive frequency band in the multi-resolution time-frequency loss is multiplied by a second weighting coefficient, wherein the first weighting coefficient is greater than the second weighting coefficient.

[0008] In some embodiments of the present invention, the plurality of resolution configurations with different window lengths and frame shifts include three sets of short-time Fourier transform configurations, each of which has a different window length and frame shift, wherein the shortest window length corresponds to the smallest frame shift and the longest window length corresponds to the largest frame shift.

[0009] In some embodiments of the present invention, the instantaneous frequency estimate is calculated by the phase difference of the short-time Fourier transform coefficients; the time spectrum after synchronous compression transform is resampled in the frequency dimension so that its dimension is consistent with the dimension of the original short-time Fourier transform coefficients.

[0010] In some embodiments of the present invention, in the calculation of the low-order cepstral envelope loss, a preset number of Mel filters are used to extract the Mel spectrum and extract the low-order cepstral coefficients of a preset order, while skipping the 0th order cepstral coefficients; the calculation of the low-order cepstral envelope loss uses at least two sets of short-time Fourier transforms with different window length configurations for feature extraction.

[0011] In some embodiments of the present invention, during the initial training rounds, the weights of the time-frequency sensing loss and the cepstral envelope loss are linearly increased from zero to a preset target value, including: In the first N rounds of training, the weight of the time-frequency sensing loss is linearly increased from zero to a first target weight, and the weight of the cepstral envelope loss is linearly increased from zero to a second target weight, wherein the first target weight is greater than the second target weight.

[0012] In some embodiments of the present invention, the speech frame features include frequency domain cepstral coefficients, pitch period, and pitch correlation; the linear prediction coefficients are calculated from the speech frame using a linear prediction analysis method.

[0013] In some embodiments of the present invention, the neural vocoder network includes a frame-level conditional network and a sample-level network; inputting the speech frame features at the current moment into the neural vocoder network to obtain a probability distribution regarding the current excitation category includes: The speech frame features are input into the frame-level conditional network, the pitch features in the speech frame features are embedded and concatenated with other features in the speech frame features, and the frame-level conditional vector is output after convolution and fully connected processing. The frame-level conditional vector is repeated in multiples of the frame length and then concatenated with the sample-by-sample input representation before being input into the sampling-level network.

[0014] In some embodiments of the present invention, the sampling-by-sampling input representation is constructed as follows: The current historical pulse code modulation sample, linear prediction value, and historical prediction error are respectively processed by μ-law encoding and embedding, mapped to embedding vectors of the same dimension, and then concatenated to obtain the sample-by-sample input representation; The sampling-level network includes multiple gated recurrent units connected in series, wherein the number of units in the first gated recurrent unit is greater than the number of units in the second gated recurrent unit, and the output of the second gated recurrent unit is processed by a fully connected layer and an activation function to output the probability distribution about the current stimulus category.

[0015] On the other hand, the present invention also provides a neural vocoder training system based on residual modeling and time-frequency perception constraints, including a processor, a memory, and a computer program or instructions stored in the memory. The processor is used to execute the computer program or instructions, and when the computer program or instructions are executed, the system implements the steps of any of the methods mentioned above.

[0016] This invention fundamentally solves the inherent defect of sample-by-sample cross-entropy loss being insensitive to spectral structure under low bit-rate quantization conditions by constructing a three-layer progressive supervision system: cross-entropy loss, time-frequency perception loss based on synchronous compression transform, and low-order cepstral envelope loss. Specifically, the time-frequency perception loss based on synchronous compression transform effectively suppresses energy diffusion in the spectrum during conventional short-time Fourier transform by refocusing the time-frequency coefficients near the instantaneous frequency trajectory. This makes the loss function more sensitive to the nonlinear distortion components introduced by low bit-rate quantization, accurately constraining the harmonic structure and time-frequency morphology of the synthesized speech. The low-order cepstral envelope loss ensures long-term timbre consistency by constraining the global spectral envelope and formant structure of the speech. The progressive weight adjustment strategy effectively avoids training oscillations caused by multi-task gradient conflicts in the early stages of training by linearly increasing the weight of the spectral loss from zero to the target value in the first few rounds, ensuring stable convergence of the joint optimization. The synergistic effect of these three components significantly improves the spectral envelope fidelity and formant structure accuracy of synthesized speech while maintaining the same inference structure and computational efficiency as existing methods. Experimental data shows that after training using the method of this invention, the local envelope distance of synthesized speech in the formant-sensitive frequency band of 200Hz~3500Hz is reduced by 46.6%, the perceived speech quality assessment score is improved by 46.5%, the subjective listening experience is clear and natural, and the mechanical and artificial feel is significantly eliminated. All improvements only apply to the training phase, without increasing the bit rate or computational complexity at the inference end. Deployed vocoders can achieve improved sound quality through retraining without modifying the inference code. This invention can be widely applied to low bit rate voice communication systems in scenarios with strictly limited bandwidth, such as satellite communication, emergency communication, and narrowband IoT.

[0017] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0018] Those skilled in the art will understand that the objectives and advantages achievable with this invention are not limited to those specifically described above, and that the above and other objectives achievable with this invention will become clearer from the following detailed description. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings: Figure 1 This is a schematic diagram illustrating the steps of a neural vocoder training method based on residual modeling and time-frequency sensing constraints in one embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram of the framework of a neural vocoder training method based on residual modeling and time-frequency sensing constraints in one embodiment of the present invention.

[0021] Figure 3 This is a flowchart of the time-frequency sensing loss module calculation based on synchronous compression transformation in one embodiment of the present invention.

[0022] Figure 4 This is a flowchart of the low-order cepstral envelope loss calculation in one embodiment of the present invention.

[0023] Figure 5(a) is a comparison of the spectral envelopes of real and synthesized speech in a 14.96 s speech segment in one embodiment of the present invention.

[0024] Figure 5(b) is a comparison of the local envelope distance of the sensitive frequency band of the 200-3500 Hz resonant peak in one embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0026] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0027] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0028] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0029] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0030] It should be emphasized here that the step markers mentioned below are not a limitation on the order of the steps, but should be understood as meaning that the steps can be executed in the order mentioned in the embodiments, or in a different order than in the embodiments, or several steps can be executed simultaneously.

[0031] To address the problems of distortion in the spectral envelope and formant structure of synthesized speech under low bit-rate quantization conditions, high-frequency harmonic attenuation and time-frequency discontinuity in transition frames, and training supervision failure due to insufficient sensitivity of conventional spectral loss to nonlinear distortion, this invention provides a neural vocoder training method based on residual modeling and time-frequency perception constraints, such as... Figure 1 and Figure 2 As shown, the method includes the following steps S101~S104: Step S101: Receive the real speech sample value at the current moment and the linear prediction coefficient of the current speech frame; calculate the linear prediction value based on the linear prediction coefficient and historical sample values; perform μ-law mapping on the difference between the real speech sample value and the linear prediction value to obtain the real excitation category; input the speech frame features at the current moment into the neural vocoder network to obtain the probability distribution of the current excitation category; perform weighted summation on the μ-law inverse mapping values ​​corresponding to each excitation category according to the probability distribution to obtain the predicted excitation; add the predicted excitation to the linear prediction value to obtain the differentiable predicted waveform.

[0032] Step S102: Based on the real waveform composed of differentiable predicted waveform and real speech sample values, jointly optimize three types of complementary constraints. The three types of complementary constraints include: excitation prediction cross-entropy loss acting on each sample point, time-frequency perception loss based on synchronous compression transform, and low-order cepstral envelope loss. The time-frequency sensing loss based on synchronous compression transform is calculated as follows: Short-time Fourier transforms are performed on the predicted and true waveforms to obtain their respective short-time Fourier transform coefficients; instantaneous frequency estimates are calculated based on the short-time Fourier transform coefficients; the time-frequency coefficients are synchronously compressed from their original positions to new positions determined by the instantaneous frequency estimates to obtain the time-frequency spectrum after synchronous compression transform; under multiple resolution configurations with different window lengths and frame shifts, the amplitude spectrum loss and logarithmic amplitude spectrum loss of the predicted and true waveforms in the synchronous compression transform domain are calculated respectively, and the losses at each resolution are summed to obtain the multi-resolution time-frequency spectrum loss; additional weights are applied to the formant sensitive frequency band to obtain the time-frequency sensing loss based on synchronous compression transform; the low-order cepstral envelope loss is calculated as follows: Mel spectra are extracted from the predicted and true waveforms respectively, and after logarithmic operations, discrete cosine transforms are performed to extract low-order cepstral coefficients; the mean square error between the low-order cepstral coefficients of the predicted and true waveforms is calculated to obtain the low-order cepstral envelope loss.

[0033] Step S103: Weight the cross-entropy loss, the time-frequency sensing loss based on synchronous compression transform, and the low-order cepstral envelope loss to construct the total loss function.

[0034] Step S104: Train the neural vocoder network using a progressive weight adjustment strategy: In the initial few rounds of training, the weights of the time-frequency perception loss and cepstral envelope loss are linearly increased from zero to a preset target value, while the weight of the cross-entropy loss remains constant. The network parameters are updated with the goal of minimizing the total loss function until training is complete. The trained neural vocoder network is then deployed to a low-bit-rate voice communication terminal. The neural vocoder network is used to reconstruct the received quantized acoustic features into a time-domain speech pulse-code modulation signal.

[0035] In step S101, a complete forward computation path is constructed from the acoustic feature input to the differentiable predictable waveform output. The core function of this step is to transform the discrete excitation class probability distribution output by the neural vocoder network into a continuously differentiable predictable waveform, thereby providing a computational basis for the subsequent gradient backpropagation of the spectral loss function.

[0036] In some embodiments, the M-order linear prediction coefficients of the current speech frame are obtained, and the linear prediction value is calculated based on these linear prediction coefficients and historical sampling values, as shown in formula (1): ; (1) in, Indicates the current time Linear prediction of the sampled values; Represents the linear prediction coefficient. ; This represents historical sample values.

[0037] Define the μ-law mapping function as This method maps the input signal to N discrete excitation categories, thus transforming the regression problem into a classification problem. Preferably, N=256. Let the set of excitation categories be denoted as . Where, any category subscript is .

[0038] In some embodiments, the true excitation category is obtained by μ-law mapping the difference between the true sampled value and the linear predicted value, as shown in formula (2): ; (2) in, Indicates the actual incentive category; This represents the current sampled value; other physical quantities have been explained above and will not be repeated here.

[0039] The speech frame features at the current moment are input into the neural vocoder network, and the network outputs a probability distribution with respect to the current activation category, i.e., for each... Output a probability value. This probability distribution reflects the network's confidence in predicting the excitation category to which the residual at the current sampling point belongs.

[0040] In some embodiments, the probability distribution is used to construct the prediction stimulus, as shown in Equation (3): ; (3) in, Indicates predicted incentives; This represents the probability distribution of the current incentive category; This represents the inverse mapping of the μ-law, which restores discrete class indices to continuous amplitudes. This weighted summation operation transforms discrete probability distributions into continuous predictions, making the entire computational path differentiable.

[0041] Finally, the predicted excitation is added to the linear predicted value to obtain a differentiable predicted waveform.

[0042] In some embodiments, the differentiable predicted waveform is calculated as shown in formula (4): ; (4) in, This represents a differentiable predicted waveform.

[0043] In some embodiments, the neural vocoder network includes a frame-level conditional network and a sample-level network. The frame-level conditional network is responsible for extracting a frame-level conditional vector from the features of the speech frames, while the sample-level network uses this conditional vector and the sample-by-sample input representation to output the excitation class probability distribution point-by-point at the sample point level.

[0044] In some embodiments, the speech frame features include frequency domain cepstral coefficients (such as Barker frequency cepstral coefficients, BFCC), pitch period, and pitch correlation. Preferably, 20-dimensional acoustic features are extracted from each frame, specifically including: 16-dimensional BFCC, 1-dimensional pitch period, and 3-dimensional pitch correlation. The linear prediction coefficients are of order 16 and are calculated from the speech frames using the Levinson-Dubin algorithm. The speech data is first processed by framing; the frame length and frame shift can be set according to the actual application scenario. Preferably, both the frame length and frame shift are 10ms.

[0045] In some embodiments, the frame-level conditional network embeds pitch features with a pitch embedding dimension of 64. The pitch embedding is then concatenated with other acoustic features to obtain an 84-dimensional vector. This vector is then passed through two one-dimensional convolutional layers with a kernel length of 3, 128 channels, and a tanh activation function. After two fully connected layers, the output dimension is 128, which serves as the frame-level conditional vector. This frame-level conditional vector is repeated in multiples of the frame length and then concatenated with the per-sample input representation before being input into the sample-level network.

[0046] In some embodiments, the sample-by-sample input representation is constructed as follows: The current historical pulse-code modulation samples, linear prediction values, and historical prediction errors are respectively encoded using μ-law (256 levels) and a differentiable embedding layer, with each embedding dimension being 128. These are then concatenated to form a 384-dimensional sample-by-sample input representation. The sample-level network includes multiple gated recurrent units (GRUs) connected in series. For example, it includes two GRU levels: GRU-A with 320 units and GRU-B with 16 units. The output of GRU-B is mapped to 256 dimensions via a fully connected layer and processed by a softmax activation function to output the probability distribution regarding the current excitation category. Due to its gated structure, the GRU network can effectively capture long-term temporal dependencies, making it suitable for processing sample-point-level sequence prediction tasks in speech signals.

[0047] In step S102, three types of complementary constraint losses are jointly calculated between the differentiable predictable waveform obtained in step S101 and the actual waveform. These three types of losses are applied at the micro (per-sampling point), meso (local time-frequency block), and macro (entire speech spectrum morphology) levels, respectively, forming a complete supervision chain from time-domain accuracy to frequency-domain fidelity.

[0048] At the sample-point level, the cross-entropy loss between the probability distribution of the stimulus class output by the neural network and the true stimulus class is calculated to measure the accuracy of the network's prediction of the stimulus class at each sample point. In some embodiments, the calculation formula is shown in Equation (5): ; (5) in, This represents the cross-entropy loss between the probability distribution of the incentive categories and the actual incentive categories; Current moment The true incentive category.

[0049] At the meso-level, this invention employs a time-frequency sensing loss module based on synchronous compressed transform (SST-TFPL module) to perform multi-resolution comparison of the time-frequency representations of the predicted waveform and the true waveform. The calculation process of this module is as follows: Figure 3 As shown.

[0050] First, short-time Fourier transforms are performed on both the predicted and actual waveforms to obtain their respective STFT coefficients. Then, instantaneous frequency estimates are calculated from the STFT coefficients.

[0051] In some embodiments, the instantaneous frequency estimation formula is as shown in equation (6): ; (6) in, Indicates time and angular frequency The instantaneous frequency estimate of the signal energy at that location; For the imaginary part operator; Indicates the STFT coefficient; This represents the partial derivative of the STFT coefficients with respect to time. This formula provides the continuous mathematical definition of instantaneous frequency estimation.

[0052] In some embodiments of the present invention, in actual discrete calculations, the partial derivatives in the above continuous formulas are... This is approximated by the phase difference between the STFT coefficients of adjacent time frames.

[0053] Then, the time-frequency coefficients are moved from their original positions. Simultaneously compress to a new location As shown in formula (7): ; (7) in, This represents the time spectrum after synchronous compression; It is the Dirac function; Represents the integral variable, and represents the time coordinate of the original STFT.

[0054] This integration operation redistributes all STFT coefficients to their corresponding positions on the frequency axis according to their instantaneous frequency estimates, resulting in a high concentration of energy near the instantaneous frequency trajectory. The synchronously compressed time spectrum exhibits higher energy concentration in the frequency dimension, more clearly revealing the harmonic structure and transient changes of the speech signal. Compared to the conventional STFT time spectrum, the synchronous compression transform effectively suppresses energy dispersion.

[0055] After obtaining the time-frequency spectrum after synchronous compression transformation, the loss is calculated under multiple resolution configurations with different window lengths and frame shifts. Let... Configure an index for the resolution. and The predicted waveform and the actual waveform are respectively at the 1st and 2nd... Synchronous compression spectrum under group configuration. Under each group configuration, the combination of amplitude spectrum loss and logarithmic amplitude spectrum loss is calculated as shown in formula (8): (8) The first term is the L1 distance of the amplitude spectrum, which represents the absolute difference in spectral amplitude under constraint; the second term is the L1 distance of the logarithmic amplitude spectrum, which represents the difference in the spectrum on a decibel scale under constraint, and is more sensitive to differences in low-energy regions (such as troughs between harmonics). These are the weighting coefficients for the logarithmic amplitude spectral loss, used to balance the magnitudes of the two types of losses. Preferably, ; To prevent numerical overflow, a small constant is usually taken as 10. -8 .

[0056] In some embodiments, the resolution configurations with different window lengths and frame shifts include three sets of short-time Fourier transform configurations, each with a different window length and frame shift, where the shortest window length corresponds to the smallest frame shift, and the longest window length corresponds to the largest frame shift. For example, the three configurations are: (window length 256, frame shift 80), (window length 512, frame shift 160), and (window length 1024, frame shift 320). The short window has higher temporal resolution, enabling it to capture transient changes in speech; the long window has higher frequency resolution, enabling it to clearly distinguish adjacent harmonics. The multi-resolution configuration constrains the time-spectrum loss on the waveform at different time-frequency scales, comprehensively ensuring transient accuracy and harmonic clarity.

[0057] The spectral loss at multiple resolutions is the sum of the losses at each resolution, as shown in formula (9): ; (9) in, Indicates time-frequency perception loss; The number configured for the resolution, preferably, .

[0058] In some embodiments, when calculating the time-frequency spectral loss, additional weights are applied to the formant-sensitive frequency band to enhance the network's focus on the time-frequency structure fidelity of this band. Specifically, the loss values ​​belonging to the formant-sensitive frequency band in the multi-resolution time-frequency loss are multiplied by a first weighting coefficient, and the loss values ​​not belonging to the formant-sensitive frequency band are multiplied by a second weighting coefficient, where the first weighting coefficient is greater than the second weighting coefficient. In one specific embodiment, the formant-sensitive frequency band is 200Hz~3500Hz, the first weighting coefficient is 1.1, and the second weighting coefficient is 1.0. This frequency band covers the range of human hearing most sensitive to and the region where speech formants are concentrated. Under low bitrate quantization conditions, the quantization distortion in this frequency band is most significant. Applying additional weights can guide the network to prioritize the use of limited model capacity to ensure the fidelity of this frequency band.

[0059] In some embodiments, the time spectrum after synchronous compression transformation is resampled in the frequency dimension to make its dimension consistent with the dimension of the original short-time Fourier transform coefficients, ensuring that the element-by-element comparison operation in loss calculation can be performed normally.

[0060] At the macroscopic level, this invention employs low-order cepstral envelope loss to constrain the consistency of the spectral envelope and formant structure of the entire speech segment. The calculation process is as follows: Figure 4 As shown.

[0061] Perceptual frequency band analysis is performed on both the predicted and actual waveforms. In some embodiments of the invention, a preset number of Mel filters are used to extract the Mel spectrum. The Mel frequency scale simulates the nonlinear perception characteristics of the human ear at different frequencies, with higher resolution in the low-frequency band and lower resolution in the high-frequency band, thus matching the characteristics of auditory perception.

[0062] Mel spectrum extraction is performed on both the predicted and actual waveforms to obtain their respective Mel spectra. The logarithm of each Mel spectrum is then taken to obtain the logarithmic Mel spectrum. Subsequently, a discrete cosine transform (DCT) is applied to the logarithmic Mel spectrum to obtain the cepstral coefficients. The DCT decorrelates the logarithmic representation of the spectrum, concentrating the spectral envelope information in the lower-order coefficients.

[0063] The mean square error between the low-order cepstral coefficients of the predicted waveform and the true waveform is calculated to obtain the low-order cepstral envelope loss.

[0064] In some embodiments, the calculation of the low-order cepstral envelope loss employs at least two sets of short-time Fourier transforms with different window length configurations for feature extraction to enhance the robustness of the envelope estimation. Specifically, 24 Mel filters are used to extract 10th-order low-order cepstral coefficients (skipping c0), employing two sets of STFT configurations with a window length of 256 / frame shift of 80° and a window length of 512 / frame shift of 160°.

[0065] In step S103, the three types of losses calculated in step S102 are weighted and summed to construct a unified total loss function, which serves as the objective function for network parameter optimization in step S104.

[0066] In some embodiments, the cross-entropy loss, the time-frequency sensing loss based on synchronous compression transform, and the low-order cepstral envelope loss are weighted and summed to construct the total loss function, as shown in Equation (10): ; (10) in, Represents the total loss function; , , These are cross-entropy loss, time-frequency sensing loss, and low-order cepstral envelope loss, respectively. These are the weighting coefficients for the time-frequency sensing loss. These are the weighting coefficients for the cepstral envelope loss.

[0067] In some embodiments, the first target weight (target weight of time-frequency perception loss) The weight is greater than the second objective weight (the objective weight of the cepstral envelope loss). This is because the time-spectral loss acts on the local time-frequency structure at the meso-level, having a more direct impact on the short-term spectral fidelity of synthesized speech, while the cepstral envelope loss, as a macro-level constraint, plays a supporting adjustment role. In a specific embodiment, , .

[0068] In step S104, with the goal of minimizing the total loss function constructed in step S103, a progressive weight adjustment strategy is used to train the neural vocoder network until training is complete, and the trained network is deployed to a low bitrate voice communication terminal.

[0069] In the initial few rounds of training, the weights of the time-frequency sensing loss and cepstral envelope loss are linearly increased from zero to a preset target value, while the weight of the cross-entropy loss remains constant (always 1). The network parameters are updated using the backpropagation algorithm with the goal of minimizing the total loss function until training is complete.

[0070] The necessity of this progressive warm-up strategy lies in the fact that, in the early stages of training, the network's fit to the residual distribution at each sampling point is insufficient, and at this time, the gradient direction of the cross-entropy loss dominates. If a spectral loss is introduced simultaneously at the beginning of training, its gradient direction may be inconsistent with that of the cross-entropy loss, leading to training oscillations or even divergence. By introducing a spectral loss in the first N rounds... and Starting from zero, the network first learns a reasonable distribution of samples by relying on cross-entropy loss to establish a stable basis for residual prediction. Then, it smoothly introduces spectral constraints to gradually align the gradient directions of the three types of losses and avoid gradient competition.

[0071] In some embodiments, during the first N epochs of training, the weights of the time-frequency perception loss are linearly increased from zero to a first target weight, and the weights of the cepstral envelope loss are linearly increased from zero to a second target weight, where the first target weight is greater than the second target weight. As in the embodiment provided above, the training epochs N are set to 10, and the first target weight... Second objective weight .

[0072] After training, the trained neural vocoder network is deployed to a low-bitrate voice communication terminal. During the inference phase, the neural vocoder network receives quantized acoustic features and outputs the excitation probability distribution point-by-point through a frame-level conditional network and a sampling-level recursive network. This distribution is then reconstructed into a time-domain speech pulse-code modulation signal via μ-law inverse mapping and linear prediction synthesis. All newly added joint optimization processes only apply to the training phase and do not change the network structure or computational flow at the inference end. Therefore, they do not increase the inference bitrate or computational complexity. Deployed vocoders can achieve improved sound quality through retraining without modifying the inference code.

[0073] In some embodiments, training employs an adaptive moment estimation optimizer (Adam) with an initial learning rate of 0.001. The batch size can be set according to hardware conditions; in one embodiment, the batch size is 8. At each epoch, loss and objective metrics are computed on the validation set to monitor the training process and avoid overfitting.

[0074] In one specific embodiment, the training data is a speech dataset with a sampling rate of 8kHz. The speech data is first processed by frame segmentation, with a frame length of 10ms (80 sampling points) and a frame shift of 10ms (no overlap). Each frame extracts 16-dimensional BFCC, 1-dimensional pitch period, and 3-dimensional pitch correlation, for a total of 20 acoustic features. Simultaneously, 16th-order LPC coefficients are used for linear prediction. The LPC coefficients are calculated from the speech frames using the Levinson-Dubin algorithm. The training hardware environment is a single GPU (such as an NVIDIA 4080 / 16G), and the total training time is approximately 80 hours.

[0075] To verify the effectiveness of the method of the present invention, speech synthesis was performed on the test set using both traditional cross-entropy training (using only cross-entropy loss) and the vocoder network trained by the method of the present invention, and the objective indicators and subjective listening test results were compared.

[0076] In the objective evaluation, the local envelope distance within the 200Hz~3500Hz frequency band was calculated using a reference speech as a benchmark. After speech synthesis using a vocoder network trained with traditional cross-entropy, the measured envelope distance was 9.17dB; while the vocoder network trained using the method of this invention reduced the envelope distance to 4.89dB under the same conditions, a relative reduction of 46.6% in local envelope error. Furthermore, the Perceptual Evaluation of Speech Quality (PESQ, wideband mode) index was calculated. PESQ scores typically range from -0.5 to 4.5, with higher values ​​indicating better perceptual quality. On the same test set, the PESQ scores of traditional methods ranged from 1.31 to 2.12, with an average of approximately 1.72; the PESQ scores obtained using the method of this invention increased to 2.35~2.68, with an average of approximately 2.52, representing an average improvement of approximately 46.5% compared to traditional methods. This indicates that the present invention can significantly reduce perceptual distortion in reconstructed speech, resulting in a clearer and more natural overall listening experience. Figure 5(a) shows a comparison of the spectral envelope of synthesized speech under different training methods. The method of the present invention is significantly better than the traditional cross-entropy training method in terms of formant structure and spectral envelope fidelity. Figure 5(b) shows a comparison of the local envelope distance of the two methods in the formant sensitive frequency band of 200-3500 Hz. The local envelope distance of the method of the present invention is reduced from 9.17 dB to 4.89 dB, a relative reduction of 46.6%.

[0077] In subjective listening tests, BFCC features were uniformly compressed using the same front-end quantization method (such as traditional three-level vector quantization, with a target bit rate of 1kbps or less), and then input into a vocoder trained using only cross-entropy loss and a vocoder trained using the joint loss method of this invention, respectively, for speech synthesis. The speech synthesized by the vocoder trained using the traditional method exhibited obvious mechanical and artificial sounds, and had inter-frame discontinuities; while the speech synthesized by the vocoder trained using the method of this invention maintained a clear formant structure and natural transitions, with significantly better spectral envelope continuity and fidelity than the traditional method. This demonstrates the improvement in synthesized sound quality achieved by the joint training method of this invention under the same low bit rate quantization feature input.

[0078] To verify the advantages of the synchronous compression transform-based time-frequency sensing loss module in this invention compared to the conventional STFT time-spectrum loss, training was conducted under the same conditions using both conventional STFT multi-resolution loss and the SST-TFPL module of this invention, with other configurations remaining consistent. Subjective listening comparisons show that the model using the SST-TFPL module produces a clearer harmonic structure in synthesized speech, retains high-frequency components more completely, and exhibits significantly better time-frequency morphological fidelity in transition frames than the conventional STFT loss. This demonstrates that the SST-TFPL module, by improving time-spectrum resolution through synchronous compression transform, can more effectively supervise the spectral reconstruction quality of low-bit-rate vocoders.

[0079] Corresponding to the above method, the present invention also provides a neural vocoder training system based on residual modeling and time-frequency perception constraints. The electronic device includes a computer device, which includes a processor and a memory. The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the electronic device implements the steps of the method described above.

[0080] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned method. The computer-readable storage medium may be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0081] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of the invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0082] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0083] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0084] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A neural vocoder training method based on residual modeling and time-frequency perception constraints, characterized in that, The method includes: The system receives the current real speech sample value and the linear prediction coefficient of the current speech frame, calculates the linear prediction value based on the linear prediction coefficient and historical sample values, performs μ-law mapping on the difference between the real speech sample value and the linear prediction value to obtain the real excitation category, inputs the speech frame features of the current moment into the neural vocoder network to obtain the probability distribution of the current excitation category, performs weighted summation of the μ-law inverse mapping values ​​corresponding to each excitation category based on the probability distribution to obtain the predicted excitation, and adds the predicted excitation to the linear prediction value to obtain a differentiable predicted waveform. Based on the differentiable predicted waveform and the real waveform formed by the real speech sample values, three types of complementary constraints are jointly optimized. These three types of complementary constraints include: excitation prediction cross-entropy loss acting on each sample point, time-frequency perception loss based on synchronous compression transform, and low-order cepstral envelope loss. The time-frequency perception loss based on synchronous compression transform is calculated as follows: short-time Fourier transforms are performed on both the predicted waveform and the real waveform to obtain their respective short-time Fourier transform coefficients; instantaneous frequency estimation is calculated based on the short-time Fourier transform coefficients; and the time-frequency coefficients are synchronously compressed from their original positions to a new position determined by the instantaneous frequency estimation to obtain the synchronously compressed transform. Time-frequency spectrum; under multiple resolution configurations with different window lengths and frame shifts, the amplitude spectrum loss and logarithmic amplitude spectrum loss of the predicted waveform and the real waveform in the synchronous compression transform domain are calculated respectively, and the losses at each resolution are summed to obtain the multi-resolution time-frequency spectrum loss; additional weights are applied to the formant sensitive frequency band to obtain the time-frequency sensing loss based on synchronous compression transform; the low-order cepstral envelope loss is calculated as follows: Mel spectrum is extracted from the predicted waveform and the real waveform respectively, and after logarithmic operation, discrete cosine transform is performed to extract the low-order cepstral coefficients. The mean square error between the low-order cepstral coefficients of the predicted waveform and the real waveform is calculated to obtain the low-order cepstral envelope loss; The total loss function is constructed by weighted summing of the cross-entropy loss, the time-frequency sensing loss based on synchronous compression transform, and the low-order cepstral envelope loss. The neural vocoder network is trained using a progressive weight adjustment strategy: in the initial few rounds of training, the weights of the time-frequency sensing loss and the cepstral envelope loss are linearly increased from zero to a preset target value, while the weight of the cross-entropy loss remains constant. The network parameters are updated with the goal of minimizing the total loss function until training is complete. The trained neural vocoder network is then deployed to a low-bit-rate voice communication terminal. The neural vocoder network is used to reconstruct the received quantized acoustic features into a time-domain speech pulse-code modulation signal.

2. The neural vocoder training method based on residual modeling and time-frequency perception constraints according to claim 1, characterized in that, Apply additional weights to the formant-sensitive frequency bands, including: The loss value belonging to the formant sensitive frequency band in the multi-resolution time-frequency loss is multiplied by a first weighting coefficient, and the loss value not belonging to the formant sensitive frequency band in the multi-resolution time-frequency loss is multiplied by a second weighting coefficient, wherein the first weighting coefficient is greater than the second weighting coefficient.

3. The neural vocoder training method based on residual modeling and time-frequency perception constraints according to claim 1, characterized in that, The resolution configurations with multiple different window lengths and frame shifts include three sets of short-time Fourier transform configurations, each with a different window length and frame shift, wherein the shortest window length corresponds to the smallest frame shift and the longest window length corresponds to the largest frame shift.

4. The neural vocoder training method based on residual modeling and time-frequency perception constraints according to claim 3, characterized in that, The instantaneous frequency estimate is calculated from the phase difference of the short-time Fourier transform coefficients; the time spectrum after synchronous compression transform is resampled in the frequency dimension so that its dimension is consistent with the dimension of the original short-time Fourier transform coefficients.

5. The neural vocoder training method based on residual modeling and time-frequency perception constraints according to claim 1, characterized in that, In the calculation of the low-order cepstral envelope loss, a preset number of Mel filters are used to extract the Mel spectrum and extract the low-order cepstral coefficients of a preset order, while skipping the 0th order cepstral coefficients; the calculation of the low-order cepstral envelope loss uses at least two sets of short-time Fourier transforms with different window length configurations for feature extraction.

6. The neural vocoder training method based on residual modeling and time-frequency sensing constraints according to claim 1, characterized in that, In the initial few rounds of training, the weights of the time-frequency sensing loss and the cepstral envelope loss are linearly increased from zero to a preset target value, including: In the first N rounds of training, the weight of the time-frequency sensing loss is linearly increased from zero to a first target weight, and the weight of the cepstral envelope loss is linearly increased from zero to a second target weight, wherein the first target weight is greater than the second target weight.

7. The neural vocoder training method based on residual modeling and time-frequency perception constraints according to claim 1, characterized in that, The speech frame features include frequency domain cepstral coefficients, pitch period, and pitch correlation; the linear prediction coefficients are calculated from the speech frames using a linear prediction analysis method.

8. The neural vocoder training method based on residual modeling and time-frequency sensing constraints according to claim 1, characterized in that, The neural vocoder network includes a frame-level conditional network and a sample-level network; Input the speech frame features at the current moment into the neural vocoder network to obtain the probability distribution of the current activation category, including: The speech frame features are input into the frame-level conditional network, the pitch features in the speech frame features are embedded and concatenated with other features in the speech frame features, and the frame-level conditional vector is output after convolution and fully connected processing. The frame-level conditional vector is repeated in multiples of the frame length and then concatenated with the sample-by-sample input representation before being input into the sampling-level network.

9. The neural vocoder training method based on residual modeling and time-frequency perception constraints according to claim 8, characterized in that, The method for constructing the sample-by-sample input representation is as follows: The current historical pulse code modulation sample, linear prediction value, and historical prediction error are respectively processed by μ-law encoding and embedding, mapped to embedding vectors of the same dimension, and then concatenated to obtain the sample-by-sample input representation; The sampling-level network includes multiple gated recurrent units connected in series, wherein the number of units in the first gated recurrent unit is greater than the number of units in the second gated recurrent unit, and the output of the second gated recurrent unit is processed by a fully connected layer and an activation function to output the probability distribution about the current stimulus category.

10. A neural vocoder training system based on residual modeling and time-frequency sensing constraints, comprising a processor, a memory, and a computer program or instructions stored in the memory, characterized in that, The processor is configured to execute the computer program or instructions, and when the computer program or instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 9.

Images