A knowledge distillation method and system of a cross-frame rate heterogeneous speech synthesis model

Abstract

The application discloses a kind of knowledge distillation methods and systems of cross frame rate heterogeneous speech synthesis model, it is related to speech synthesis and depth learning technical field.The present application includes: generating teacher speech synthesis audio using teacher model single inference, extracting teacher mel spectrum with the acoustic parameter of student speech synthesis model after resampling;Frame rate ratio is calculated and linear interpolation upsampling is independently carried out to each sample, frame rate alignment is realized;In VITS segmentation training, the same segmentation starting position and segmentation frame number as student speech synthesis model are used to slice the teacher mel spectrum after alignment simultaneously, and effective frame mask is generated;The segmentation level distillation loss after mask weighting is weighted with the standard training loss of student speech synthesis model Sum, jointly optimize student speech synthesis model parameter.The present application realizes the effective acoustic knowledge migration between heterogeneous frame rate TTS model, ensures the compatibility of distillation training and segmentation training paradigm.

Description

Technical Field

[0001] This invention relates to the fields of speech synthesis and deep learning technology, specifically to a knowledge distillation method and system for cross-frame-rate heterogeneous speech synthesis models. Background Technology

[0002] In the field of end-to-end speech synthesis, VITS (Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech), as an end-to-end TTS framework combining variational autoencoders and adversarial learning, can directly generate speech waveforms from text. It exhibits excellent performance in both naturalness and real-time performance, and has become one of the mainstream solutions for lightweight speech synthesis. VITS employs a random segmentation training paradigm, randomly extracting fixed-length segments from the complete spectrum during training to calculate the loss, thus balancing training efficiency and generation quality. It is widely used in resource-constrained embedded devices and real-time interactive scenarios.

[0003] Meanwhile, large-scale speech synthesis models, such as CosyVoice3, far surpass traditional lightweight models in timbre fidelity, prosodic naturalness, and cross-linguistic capabilities by modeling semantic tokens using large language models (LLMs) and combining flow matching with HiFT vocoders. However, such large models typically have hundreds of millions to billions of parameters, resulting in high inference latency and making them unsuitable for deployment on embedded devices or real-time interactive scenarios.

[0004] To alleviate the performance gap between large and lightweight models, knowledge distillation, as an effective model compression and knowledge transfer method, is considered an important technical path to improve the quality of lightweight TTS models. However, when applying knowledge distillation to end-to-end speech synthesis models, the Mel-spectral frame rates of the teacher and student models often differ. For example, CosyVoice3 uses a 50Hz frame rate, while Piper VITS uses approximately 86.13Hz. This frame rate difference results in the same speech segment corresponding to different numbers of spectral frames on the teacher and student sides, making traditional frame-by-frame distillation methods unsuitable. Existing methods either directly ignore the frame rate difference, rendering frame-by-frame comparison ineffective, or use sentence scaling followed by truncation, leading to insufficient alignment accuracy at boundaries, inaccurate distillation loss calculations, and temporal shifts in the acoustic supervision signals received by the student model, severely affecting the naturalness and prosodic accuracy of the synthesized speech.

[0005] To this end, this invention proposes a knowledge distillation method and system for cross-frame-rate heterogeneous speech synthesis models. Summary of the Invention

[0006] The purpose of this invention is to provide a knowledge distillation method and system for cross-frame-rate heterogeneous speech synthesis models to solve the problems mentioned in the background art.

[0007] According to a first aspect of the present invention, in order to achieve the above-mentioned objective, the present invention provides the following technical solution: a knowledge distillation method for cross-frame-rate heterogeneous speech synthesis models, comprising the following steps: Acquire training text, call the teacher speech synthesis model to perform a single inference on the training text, and output the teacher audio waveform corresponding to the training text; The teacher's audio waveform is resampled to the target sampling rate of the student speech synthesis model, and the Mel spectrum extraction parameters of the student speech synthesis model are used to extract the teacher's Mel spectrum from the resampled teacher's audio waveform, and the sample-by-sample actual effective length of the teacher's Mel spectrum is output. Based on the teacher frame rate of the teacher speech synthesis model and the student frame rate of the student speech synthesis model, the frame rate ratio is calculated. Based on the actual effective length per sample and the frame rate ratio, the target number of frames is calculated per sample. Linear interpolation is used to upsample the teacher Mel spectrum to the target number of frames in the time dimension to obtain the aligned teacher Mel spectrum. During the segmented random training process of the student speech synthesis model, the student speech synthesis model randomly selects a starting position from the student Mel spectrum, and extracts a fixed number of student spectrum segments based on the selected starting position, and records the selected starting position and the fixed number of frames. Based on the recorded starting position and the fixed number of frames, a teacher spectrum segment is extracted from the aligned teacher Mel spectrum, and the teacher spectrum segment and the student spectrum segment correspond strictly to each other on the time axis. The student spectrum segment and the teacher spectrum segment are length aligned, and the element-wise distillation loss between them is calculated. The difference between the aligned effective frame length of the teacher Mel spectrum and the truncation start position is calculated to obtain the number of effective frames in the segment. An effective frame mask is generated based on the number of effective frames in the segment. The element-wise distillation loss is weighted with the effective frame mask, and the gradient contribution of the filling frame region is masked to obtain the mask-weighted distillation loss. The total training loss is obtained by weighting the distillation loss after masking and summing it with the standard training loss of the student speech synthesis model according to a preset weight coefficient. The network parameters of the student speech synthesis model are then updated by backpropagation based on the total training loss.

[0008] Furthermore, the Mel spectrum extraction parameters include filter length, step length, window length, and number of Mel bands, and the extraction parameters of the teacher's Mel spectrum are completely consistent with the extraction parameters of the student's Mel spectrum; The teacher's audio waveform and the teacher's Mel spectrum are generated by the same inference process and maintain temporal consistency, and the teacher's Mel spectrum and the student's Mel spectrum output by the student speech synthesis model are in the same acoustic parameter domain.

[0009] Furthermore, the frame rate ratio For the student frame rate With the teacher frame rate The ratio is calculated as follows: The step-by-sample calculation of the target frame count includes: for each sample in the batch, rounding down the product of its actual effective frame length and the frame rate ratio to obtain the target frame count for that sample. In the formula, The first in the batch The target number of frames per sample, i.e. the number of frames that should be reached after the teacher upsamples the Mel spectrum to the student frame rate; This is the actual effective frame length of the teacher's Mel spectrum for this sample, i.e., the original number of frames excluding padding frames; This refers to the frame rate ratio, specifically the student frame rate. With teacher frame rate The ratio; This indicates rounding to the nearest integer. When the frame rate ratio is 1, skip the linear interpolation and directly copy the teacher's Mel spectrum.

[0010] Furthermore, the segmented random training refers to the random segmentation and slicing training of the VITS model, where the starting position of the slicing is... The parameter, the fixed number of frames is parameter.

[0011] Furthermore, the element-wise distillation loss includes at least one of L1 loss, L2 loss, or Smooth L1 loss; The effective frame mask is A binary mask of type.

[0012] Furthermore, the calculation method for the mask-weighted distillation loss is as follows: The element-wise distillation loss is multiplied element-wise by the effective frame mask, the product is summed over the effective frame region, and then divided by the product of the total number of effective frames and the number of Mel bands to obtain the mask-weighted distillation loss, as shown below: In the formula, The element-wise distillation loss tensor has the following shape: ,in For batch size, For Mel band number, For time frames; The effective frame mask tensor has the following shape: The binary tensor, with the effective frame position taking the value The fill frame position is set to a value. During calculation, the broadcast mechanism is used to extend to... Same dimensions; The total number of valid frames, i.e., all values ​​in the mask. The sum of the number of elements; This represents the summation of all element-wise products over the batch dimension, Mel band dimension, and time frame dimension; When the effective frame mask is all zeros, zero loss is returned.

[0013] Furthermore, the standard training loss includes adversarial generator loss, feature matching loss, Mel spectrum reconstruction loss, duration prediction loss, and KL divergence loss; the total training loss is calculated using the following formula: in, To combat generator loss, For feature matching loss, For Mel spectrum reconstruction loss, To predict losses over time, For KL divergence loss, The distillation loss is weighted by the mask. , and These are the weighting coefficients for the corresponding losses.

[0014] Furthermore, the distillation loss is configured with a delayed start mechanism, which includes: in the initial stage of training, setting the weight coefficient of the distillation loss to zero, and then enabling the distillation loss to participate in the weighted summation after the training steps reach a preset number of distillation start steps.

[0015] Furthermore, the teacher's speech synthesis model is a large-scale speech synthesis model, the student's speech synthesis model is a lightweight end-to-end speech synthesis model, and the teacher's frame rate is different from the student's frame rate.

[0016] According to a second aspect of the present invention, the present invention provides a knowledge distillation system for cross-frame-rate heterogeneous speech synthesis models, used to implement the knowledge distillation method for cross-frame-rate heterogeneous speech synthesis models described in the first aspect, comprising: The teacher audio and Mel spectrum generation module is used to acquire training text and call the teacher speech synthesis model to perform a single inference, output the teacher audio waveform, resample the teacher audio waveform to the target sampling rate of the student speech synthesis model, extract the teacher Mel spectrum using the Mel spectrum extraction parameters of the student speech synthesis model, and record the sample-by-sample actual effective length of the teacher Mel spectrum. The teacher audio waveform and the teacher Mel spectrum are generated by the same inference process and maintain temporal consistency. The Mel spectrum alignment module is used to calculate the frame rate ratio based on the teacher frame rate and the student frame rate, calculate the target number of frames per sample based on the actual effective length per sample and the frame rate ratio, and use linear interpolation to upsample the teacher Mel spectrum to the target number of frames in the time dimension to obtain the aligned teacher Mel spectrum. The segmented distillation loss calculation module is used to extract teacher spectrum segments from the aligned teacher Mel spectrum based on the starting position of the student spectrum segment and a fixed number of frames during the segmented random training process of the student speech synthesis model. The module calculates element-wise distillation loss for the length-aligned student spectrum segment and the teacher spectrum segment, and uses an effective frame mask to mask the gradient contribution of the filling frame region to obtain the mask-weighted distillation loss. The multi-loss joint optimization training module is used to sum the mask-weighted distillation loss with the standard training loss of the student speech synthesis model according to preset weight coefficients to obtain the total training loss, and to update the network parameters of the student speech synthesis model based on the total training loss through backpropagation.

[0017] This invention has at least the following beneficial effects: 1. This invention solves the technical obstacle of not being able to directly perform frame-by-frame distillation between teacher and student models due to the difference in Mel spectrum frame rates by independently calculating the target frame number per sample and using linear interpolation for frame rate upsampling. Regardless of whether the frame rate ratio between the teacher and student models is an integer, the teacher Mel spectrum can be mapped to the student frame rate space through accurate sample-by-sample alignment, so that the distillation loss is accurately aligned in the time dimension. This enables effective acoustic-level knowledge transfer between heterogeneous speech synthesis models with different architectures and parameter scales.

[0018] 2. This invention injects the rich acoustic details and prosodic information of a large-scale teacher model into a lightweight student model in the form of distillation supervision. While maintaining the original lightweight size and inference efficiency, the student model achieves significant improvements in the timbre consistency, prosodic naturalness, and spectral detail richness of its synthesized speech, effectively narrowing the quality gap with the teacher model. This enables the lightweight model to output synthesis results close to those of the large model in resource-constrained scenarios.

[0019] 3. This invention employs a segmented synchronous slicing mechanism, ensuring that the teacher's and student's spectral segments are extracted at exactly the same time position. This guarantees that the distillation loss, along with the adversarial generation loss, feature matching loss, Mel reconstruction loss, duration prediction loss, and KL divergence loss of the VITS model, are calculated on the same time segment. The consistency of the multi-loss optimization directions avoids temporal offset conflicts between the distilled and reconstructed signals, resulting in a stable training process and convergence characteristics superior to whole-sentence-level distillation schemes.

[0020] 4. By accurately calculating the number of valid frames within a segment based on the aligned effective frame length and the segment start position, and generating a binary mask to perform gradient masking on the padding frame region, this invention accurately eliminates the interference of zero-padding frames on the distillation loss during batch training. Distillation supervision is contributed only by real speech frames, avoiding the dilution of the loss mean and misleading of the training direction by invalid frames, and significantly improving the signal quality of distillation supervision.

[0021] 5. The distillation loss type, loss weight coefficient, and delayed start steps of this invention are all configurable parameters, which can be flexibly adjusted according to different combinations of teacher and student models, different training data characteristics, and different application scenarios. The delayed start mechanism allows the student model to converge its basic acoustic capabilities in the initial training stage before introducing distillation supervision, further improving the stability and adaptability of the training.

[0022] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the overall process of the method described in this invention; Figure 2 This is a schematic diagram of the system described in this invention; Figure 3 This is a flowchart of the Mel spectrum frame rate alignment and segmentation slicing process in this invention; Figure 4 This is a schematic diagram of the distillation loss calculation process of the present invention; Figure 5 This is a schematic diagram of the multi-loss joint optimization training of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] Example 1: Explanation of related terms: VITS (Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech) is an end-to-end TTS framework that combines variational autoencoders and adversarial learning, enabling direct generation of speech waveforms from text.

[0027] TTS (Text-to-Speech): A technology that uses computers to convert text information into natural speech output, belonging to the field of speech synthesis.

[0028] Knowledge Distillation: A model compression method that transfers knowledge from a large-scale teacher model to a lightweight student speech synthesis model. By having the student speech synthesis model learn the output distribution or intermediate representation of the teacher model, it improves the performance of the student speech synthesis model while maintaining a small model size.

[0029] CosyVoice3 is a large-scale speech synthesis model that uses a large language model (LLM) to model semantic tokens and combines flow matching with a HiFT vocoder to achieve speech synthesis. It performs well in terms of timbre fidelity, prosodic naturalness, and cross-linguistic capabilities.

[0030] Mel spectrum: a time-frequency representation based on the characteristics of human auditory perception. It is obtained by converting the audio signal to the frequency domain through short-time Fourier transform, and then weighted and summing it through a Mel frequency scale filter bank. It is widely used in speech synthesis, speech recognition and other fields.

[0031] Random Segment Slicing: A training paradigm that randomly selects a starting position from the complete spectrum and extracts a fixed-length segment for loss calculation during the training process. The VITS model uses this training method.

[0032] LLM (Large Language Model): A large-scale pre-trained language model based on the Transformer architecture. In this invention, CosyVoice3 uses LLM to model semantic tokens.

[0033] Please see Figure 1 This invention provides a technical solution: a knowledge distillation method for cross-frame-rate heterogeneous speech synthesis models, comprising the following steps: S1. Obtain the training text, call the teacher speech synthesis model to perform a single inference on the training text, and output the teacher audio waveform corresponding to the training text, as follows: Add the prompt prefix required by the teacher's speech synthesis model (such as "You are a helpful assistant.<|endofprompt|>") to the input training text, and call the CosyVoice3 cross-lingual inference interface (inference_cross_lingual) to generate the speech waveform; S2. Resample the teacher's audio waveform to the target sampling rate of the student speech synthesis model, and extract the teacher's Mel spectrum from the resampled teacher's audio waveform using the Mel spectrum extraction parameters of the student speech synthesis model. Record the actual effective length of the teacher's Mel spectrum per sample, as follows: Step 201: Resample the generated teacher speech waveform to the sampling rate of the student speech synthesis model, specifically 22050Hz. Then, use the Mel spectrum extraction parameters (filter_length=1024, hop_length=256, win_length=1024, n_mels=80) of the student speech synthesis model to calculate the teacher Mel spectrum. This design makes the teacher Mel spectrum consistent with the student Mel spectrum in the acoustic parameter domain, avoiding additional errors in cross-domain comparison. It should be noted that the teacher's audio waveform and the teacher's Mel spectrum are generated by the same inference process and maintain temporal consistency; Step 202: Save the teacher's Mel spectrum corresponding to each text as an independent file (e.g., .npy format) and generate a manifest file (manifest.json) to record the meta-information of each sample, such as the text, audio path, Mel spectrum path, frame number, and teacher frame rate; S3. Based on the teacher frame rate of the teacher speech synthesis model and the student frame rate of the student speech synthesis model, calculate the frame rate ratio. Based on the actual effective length per sample and the frame rate ratio, calculate the target number of frames per sample. Use linear interpolation to upsample the teacher Mel spectrum to the target number of frames in the time dimension to obtain the aligned teacher Mel spectrum, as follows: Step 301: Frame Rate Ratio Calculation During initialization, the frame rate is determined based on the teacher's frame rate (e.g., 50Hz) and the student's frame rate (student sampling rate / student hop_length, e.g.) Calculate the frame rate ratio: For example, ; Step 302: Per-sample frame rate alignment For each sample in the batch, calculate the target number of aligned frames for that sample based on the actual effective length of its teacher mel spectrum (teacher_mel_lengths) and the frame rate ratio: In the formula, The first in the batch The target number of frames per sample, i.e. the number of frames that should be reached after the teacher upsamples the Mel spectrum to the student frame rate; This is the actual effective frame length of the teacher's Mel spectrum for this sample, i.e., the original number of frames excluding padding frames; The frame rate ratio; This indicates rounding to the nearest integer. In the formula, The first in the batch The target number of frames per sample, i.e. the number of frames that should be reached after the teacher upsamples the Mel spectrum to the student frame rate; This is the actual effective frame length of the teacher's Mel spectrum for this sample, i.e., the original number of frames excluding padding frames; This refers to the frame rate ratio, specifically the student frame rate. With teacher frame rate The ratio, for example, when the teacher's frame rate is 50Hz and the student's frame rate is approximately 86.13Hz, is approximately [value missing]. ; This indicates rounding to the nearest integer. The teacher's Mel spectrum is upsampled from frame t_len to frame t_target_b in the time dimension using linear interpolation (F.interpolate, mode='linear', align_corners=False). This sample-by-sample processing ensures that samples of different lengths are precisely aligned, rather than using a globally uniform scaling factor. Step 303: Valid Frame Mask Generation Calculate the effective frame length (aligned_lengths) after alignment for each sample, subtract the segment start position (ids_slice) to obtain the number of effective frames within the segment, and generate a binary mask of type sequence_mask to mask the gradient contribution of the filling frame region in the distillation loss calculation; S4. During the segmented random training process of the student speech synthesis model, the student speech synthesis model randomly selects a starting position from the student Mel spectrum, and extracts a fixed number of student spectrum segments based on the selected starting position. The starting position and the fixed number of frames are recorded, as follows: In VITS (Segmented Random Training), the student speech synthesis model randomly selects start and end positions (ids_slice) from the complete spectrum to extract a fixed number of frames (segment_frames) to calculate the loss. In this embodiment, after aligning the teacher's spectrum, the same ids_slice and segment_frames parameters as the student's are used to segment the aligned teacher's Mel spectrum (by calling the slice_segments function) to ensure that the teacher's spectrum segments and the student's spectrum segments strictly correspond on the time axis. like Figure 3 As shown, the Mel spectrum frame rate alignment and segmentation process described in steps S3 and S4 is specifically demonstrated, with the complete teacher Mel spectrum as input. → Calculate the number of target frames per sample → Upsample to student frame rate using per-sample linear interpolation → Synchronously slice based on student training ids_slice and segment_frames → Output teacher spectrum segment and effective frame mask ; S5. Based on the recorded starting position and fixed number of frames, extract the teacher spectrum segment from the aligned teacher Mel spectrum. The teacher spectrum segment and the student spectrum segment are strictly corresponding on the time axis. In step S5, using the same idsslice and segmentframes parameters as in step S4, the slicesegments function is called to extract teacher spectrum segments from the aligned teacher Mel spectrum. S6. Align the student and teacher spectrum segments by length, calculate the element-wise distillation loss between them, calculate the difference between the aligned effective frame length of the teacher Mel spectrum and the truncation start position, and truncate the difference to the interval using clamping operations. Within, the number of valid frames within the segment is obtained. Then generate a sample of length for each sample. A binary mask sequence, where the first... The value at each position is... This indicates the valid frame region; the values ​​at other locations are... This indicates the filling frame region; specifically, when the length of the aligned effective frame is less than or equal to the truncation start position, the difference is clamped. The mask sequence is all This sample contributes zero distillation loss; when the effective alignment frame length is greater than the truncation start position, the mask sequence contains indivual and indivual The element-wise distillation loss is multiplied element-wise by the effective frame mask to mask the gradient contribution of the filled frame region. The average loss value of the effective frame region is then calculated to obtain the mask-weighted distillation loss, as follows: Step 601: Length Safety Check If there is still a slight difference in length between the student and teacher spectrum segments in the time dimension (due to the non-integer frame rate ratio), truncate to the shorter length. Step 602: Loss Function Calculation Supports multiple loss types: - L1 loss (default): - L2 / MSE loss: - Smooth L1 / Huber Loss: More robust to outliers Step 603: Mask Weighting Multiply the distillation loss by the effective frame mask generated in step 303, and calculate the mean loss only for the effective frame region: In the formula, Let the element-wise distillation loss tensor be of shape as follows: ,in For batch size, For Mel band number, For time frames; The effective frame mask tensor has the following shape: The binary tensor, with the effective frame position taking the value The fill frame position is set to a value. During calculation, the broadcast mechanism is used to extend to... Same dimensions; The total number of valid frames, i.e., all values ​​in the mask. The sum of the number of elements; This represents the summation of all element-wise products over the batch dimension, Mel band dimension, and time frame dimension; When the mask is all zeros (the teacher segment is completely invalid), return zero loss to avoid numerical anomalies; Figure 4 A detailed demonstration of the distillation loss calculation process: Student Mel spectrum fragment With Teacher Mel's Spectrum Fragment → Length truncation and alignment → Calculate element-wise loss (L1 / L2 / Huber) → Multiply by effective frame mask → Calculate mask region mean → Multiply by weighting coefficient → Obtain the final distillation loss; S7. The mask-weighted distillation loss is summed with the standard training loss of the student speech synthesis model using preset weighting coefficients to obtain the total training loss. The network parameters of the student speech synthesis model are then updated based on the total training loss through backpropagation, as follows: The distillation loss is jointly optimized with the VITS standard training loss, referring to... Figure 5 The total training loss is: in: : Adversarial generator loss; Feature matching loss; Mel spectrum reconstruction loss (L1; : Duration-based predicted loss; KL divergence loss; Distillation loss (calculated by module 103); (Default 35) (Default 1.0) (Default 10.0) represents the hyperparameters for each loss weight; Distillation losses also support delayed start-up mechanisms ( This means that distillation is not enabled in the initial training phase, and distillation supervision is introduced after the model converges its basic acoustic capabilities to improve training stability. The value can be set proportionally based on the total number of training steps; an empirical range is the total number of training steps. When the total number of training steps is When taking a step, the corresponding value is to Step 1; further, the delayed start mechanism can be combined with Mel-spectrum reconstruction loss. Adaptive determination of convergence state: continuous monitoring during training. In continuous Step size (e.g.) The relative rate of decline within the sliding window, when the rate of decline is lower than a preset convergence threshold. (like When the basic acoustic reconstruction capability of the student model is determined to be initially stable, distillation loss supervision is automatically enabled. Furthermore, after enabling distillation, a gradual heating strategy can be adopted, adjusting the distillation weighting coefficients during the preheating period. from Increment to the target value using linear or cosine scheduling (e.g.) This avoids the sudden introduction of distillation supervision causing shocks to the training process; Figure 5 The diagram illustrates the joint optimization training using multiple losses, specifically showing the composition and calculation relationships of each loss term in the training of the student speech synthesis model: The forward propagation of the student speech synthesis model (SynthesizerTrn) yields... , , , Calculate separately (Countering losses) (Feature matching loss) (Meyer's reconstruction losses) (Predicted loss over time) (KL divergence loss) (Distillation loss) → Weighted summation yields the total loss → Backpropagation updates the parameters of the student speech synthesis model.

[0034] It should be noted that the following alternative solutions are merely optional extensions of the present invention and do not constitute a limitation on the essential technical features of this embodiment: This embodiment uses linear interpolation (F.interpolate, mode='linear') for frame rate upsampling. Alternative methods include nearest neighbor interpolation (mode='nearest'), cubic spline interpolation (mode='cubic'), or non-linear alignment based on a learnable upsampling network (such as transposed convolution). Linear interpolation is sufficiently accurate in the spectral domain, but other interpolation methods may provide better spectral fidelity at higher frame rate difference ratios. This embodiment does not impose specific limitations and the choice can be made according to the specific circumstances.

[0035] Furthermore, this embodiment focuses on Mel-frequency distillation. Based on Mel-frequency distillation, latent z-p space distillation and duration prediction distillation can be introduced to form a multi-level distillation framework. Each level calculates the loss independently and is weighted with adjustable weights. Latent space distillation supports projection adaptation between different channel dimensions, while duration distillation focuses on relative duration patterns (normalized duration differences rather than absolute values).

[0036] Example 2: like Figure 2 As shown, this embodiment provides a knowledge distillation system for cross-frame-rate heterogeneous speech synthesis models, used to implement the knowledge distillation method for cross-frame-rate heterogeneous speech synthesis models described in Embodiment 1, including: The teacher audio and Mel spectrum generation module is used to acquire training text and call the teacher speech synthesis model to perform a single inference, output the teacher audio waveform, resample the teacher audio waveform to the target sampling rate of the student speech synthesis model, extract the teacher Mel spectrum using the Mel spectrum extraction parameters of the student speech synthesis model, and record the actual effective length of the teacher Mel spectrum per sample. The teacher audio waveform and the teacher Mel spectrum are generated by the same inference process and maintain temporal consistency. The Mel spectrum alignment module is used to calculate the frame rate ratio based on the teacher's frame rate and the student's frame rate. Based on the actual effective length per sample and the frame rate ratio, it calculates the target number of frames per sample. Linear interpolation is used to upsample the teacher's Mel spectrum to the target number of frames in the time dimension to obtain the aligned teacher's Mel spectrum. The segmented distillation loss calculation module is used in the segmented random training process of the student speech synthesis model. Based on the starting position of the student spectrum segment and a fixed number of frames, the module extracts the teacher spectrum segment from the aligned teacher Mel spectrum. It calculates the element-wise distillation loss for the length-aligned student spectrum segment and the teacher spectrum segment, and uses an effective frame mask to mask the gradient contribution of the filling frame region to obtain the mask-weighted distillation loss. The multi-loss joint optimization training module is used to sum the mask-weighted distillation loss with the standard training loss of the student speech synthesis model according to preset weight coefficients to obtain the total training loss, and to update the network parameters of the student speech synthesis model based on backpropagation of the total training loss.

[0037] Furthermore, the Mel spectrum alignment module includes a frame rate ratio calculation submodule for calculating the frame rate ratio based on the teacher's frame rate and the student's frame rate.

[0038] In the description of this specification, the references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0039] The foregoing has shown and described the basic principles, main features, and advantages of this application. Those skilled in the art should understand that this application is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this application. Various changes and modifications can be made to this application without departing from the spirit and scope thereof, and all such changes and modifications fall within the scope of the claims of this application.

Claims

1. A knowledge distillation method for cross-frame-rate heterogeneous speech synthesis models, characterized in that, Includes the following steps: Acquire training text, call the teacher speech synthesis model to perform a single inference on the training text, and output the teacher audio waveform corresponding to the training text; The teacher's audio waveform is resampled to the target sampling rate of the student speech synthesis model, and the Mel spectrum extraction parameters of the student speech synthesis model are used to extract the teacher's Mel spectrum from the resampled teacher's audio waveform, and the sample-by-sample actual effective length of the teacher's Mel spectrum is output. Based on the teacher frame rate of the teacher speech synthesis model and the student frame rate of the student speech synthesis model, the frame rate ratio is calculated. Based on the actual effective length per sample and the frame rate ratio, the target number of frames is calculated per sample. Linear interpolation is used to upsample the teacher Mel spectrum to the target number of frames in the time dimension to obtain the aligned teacher Mel spectrum. During the segmented random training process of the student speech synthesis model, the student speech synthesis model randomly selects a starting position from the student Mel spectrum, and extracts a fixed number of student spectrum segments based on the selected starting position, and records the selected starting position and the fixed number of frames. Based on the recorded starting position and the fixed number of frames, a teacher spectrum segment is extracted from the aligned teacher Mel spectrum, and the teacher spectrum segment and the student spectrum segment correspond strictly to each other on the time axis. The student spectrum segment and the teacher spectrum segment are length aligned, and the element-wise distillation loss between them is calculated. The difference between the aligned effective frame length of the teacher Mel spectrum and the truncation start position is calculated to obtain the number of effective frames in the segment. An effective frame mask is generated based on the number of effective frames in the segment. The element-wise distillation loss is weighted with the effective frame mask, and the gradient contribution of the filling frame region is masked to obtain the mask-weighted distillation loss. The total training loss is obtained by weighting the distillation loss after masking and summing it with the standard training loss of the student speech synthesis model according to a preset weight coefficient. The network parameters of the student speech synthesis model are then updated by backpropagation based on the total training loss.

2. The knowledge distillation method for a cross-frame-rate heterogeneous speech synthesis model according to claim 1, characterized in that: The Mel spectrum extraction parameters include filter length, step length, window length, and Mel band number. The extraction parameters of the teacher's Mel spectrum are completely consistent with the extraction parameters of the student's Mel spectrum. The teacher's audio waveform and the teacher's Mel spectrum are generated by the same inference process and maintain temporal consistency, and the teacher's Mel spectrum and the student's Mel spectrum output by the student speech synthesis model are in the same acoustic parameter domain.

3. The knowledge distillation method for a cross-frame-rate heterogeneous speech synthesis model according to claim 1, characterized in that: The frame rate ratio For the student frame rate With the teacher frame rate The ratio is calculated as follows: The step-by-sample calculation of the target frame count includes: for each sample in the batch, rounding down the product of its actual effective frame length and the frame rate ratio to obtain the target frame count for that sample. In the formula, The first in the batch The target number of frames per sample, i.e. the number of frames that should be reached after the teacher upsamples the Mel spectrum to the student frame rate; This is the actual effective frame length of the teacher's Mel spectrum for this sample, i.e., the original number of frames excluding padding frames; This refers to the frame rate ratio, specifically the student frame rate. With teacher frame rate The ratio; This indicates rounding to the nearest integer. When the frame rate ratio is 1, skip the linear interpolation and directly copy the teacher's Mel spectrum.

4. The knowledge distillation method for a cross-frame-rate heterogeneous speech synthesis model according to claim 1, characterized in that: In step S4, the segmented random training refers to the random segmentation and slicing training of the VITS model, and the starting position of the slicing is... The parameter, the fixed number of frames is parameter.

5. The knowledge distillation method for a cross-frame-rate heterogeneous speech synthesis model according to claim 1, characterized in that: The element-by-element distillation loss includes at least one of L1 loss, L2 loss, or Smooth L1 loss; The effective frame mask is A binary mask of type.

6. The knowledge distillation method for a cross-frame-rate heterogeneous speech synthesis model according to claim 1, characterized in that: The method for calculating the distillation loss after mask weighting is as follows: The element-wise distillation loss is multiplied element-wise by the effective frame mask, the product is summed over the effective frame region, and then divided by the product of the total number of effective frames and the number of Mel bands to obtain the mask-weighted distillation loss, as shown below: In the formula, The element-wise distillation loss tensor has the following shape: ,in For batch size, For Mel band number, For time frames; The effective frame mask tensor has the following shape: The binary tensor, with the effective frame position taking the value The fill frame position is set to a value. During calculation, the broadcast mechanism is used to extend to... Same dimensions; The total number of valid frames, i.e., all values ​​in the mask. The sum of the number of elements; This represents the summation of all element-wise products over the batch dimension, Mel band dimension, and time frame dimension. When the effective frame mask is all zeros, zero loss is returned.

7. The knowledge distillation method for a cross-frame-rate heterogeneous speech synthesis model according to claim 1, characterized in that: The standard training loss includes adversarial generator loss, feature matching loss, Mel spectrum reconstruction loss, duration prediction loss, and KL divergence loss; the total training loss is calculated using the following formula: in, To combat generator loss, For feature matching loss, For Mel spectrum reconstruction loss, To predict losses over time, For KL divergence loss, The distillation loss is weighted by the mask. , and These are the weighting coefficients for the corresponding losses.

8. The knowledge distillation method for a cross-frame-rate heterogeneous speech synthesis model according to claim 1, characterized in that: The distillation loss is configured with a delayed start mechanism, which includes: in the initial stage of training, setting the weight coefficient of the distillation loss to zero, and then enabling the distillation loss to participate in the weighted summation after the training steps reach a preset number of distillation start steps.

9. The knowledge distillation method for a cross-frame-rate heterogeneous speech synthesis model according to claim 1, characterized in that: The teacher's speech synthesis model is a large-scale speech synthesis model, and the student's speech synthesis model is a lightweight end-to-end speech synthesis model. The teacher's frame rate is different from the student's frame rate.

10. A knowledge distillation system for a cross-frame-rate heterogeneous speech synthesis model, used to implement the knowledge distillation method for the cross-frame-rate heterogeneous speech synthesis model as described in any one of claims 1 to 9, characterized in that, include: The teacher audio and Mel spectrum generation module is used to acquire training text and call the teacher speech synthesis model to perform a single inference, output the teacher audio waveform, resample the teacher audio waveform to the target sampling rate of the student speech synthesis model, extract the teacher Mel spectrum using the Mel spectrum extraction parameters of the student speech synthesis model, and record the sample-by-sample actual effective length of the teacher Mel spectrum. The teacher audio waveform and the teacher Mel spectrum are generated by the same inference process and maintain temporal consistency. The Mel spectrum alignment module is used to calculate the frame rate ratio based on the teacher frame rate and the student frame rate, calculate the target number of frames per sample based on the actual effective length per sample and the frame rate ratio, and use linear interpolation to upsample the teacher Mel spectrum to the target number of frames in the time dimension to obtain the aligned teacher Mel spectrum. The segmented distillation loss calculation module is used to extract teacher spectrum segments from the aligned teacher Mel spectrum based on the starting position of the student spectrum segment and a fixed number of frames during the segmented random training process of the student speech synthesis model. The module calculates element-wise distillation loss for the length-aligned student spectrum segment and the teacher spectrum segment, and uses an effective frame mask to mask the gradient contribution of the filling frame region to obtain the mask-weighted distillation loss. The multi-loss joint optimization training module is used to sum the mask-weighted distillation loss with the standard training loss of the student speech synthesis model according to preset weight coefficients to obtain the total training loss, and to update the network parameters of the student speech synthesis model based on the total training loss through backpropagation.

Images