Text-to-audio generation distillation method based on balanced signal-to-noise ratio perception

Abstract

The invention belongs to the field of audio generation and processing, and relates to a text-to-audio generation distillation method based on balanced signal-to-noise ratio perception, which comprises the following steps: acquiring a text and inputting the text into a trained student model to obtain an audio; the training process of the student model comprises the following steps: initializing the student model according to the teacher model; obtaining a text feature tau and an audio feature z0 according to the text and audio data; inputting the characteristics tau and z0 into the teacher model, and inputting the output of the teacher model and the characteristics tau into the student model; calculating a BSA loss function value according to the output of the teacher model and the student model, updating the parameters of the student model, and when the student model reaches a preset training frequency, if the maximum distillation frequency is reached, obtaining a trained student model; otherwise, setting parameters of the teacher model according to the student model and carrying out next distillation; according to the method, the BSA loss function is combined, knowledge transmission between the teacher model and the student model is optimized, and the distillation process can keep high stability and accuracy under different signal-to-noise ratio conditions.

Description

Technical Field

[0001] The present invention belongs to the field of audio generation and processing, and relates to a text-to-audio generation distillation method based on balanced signal-to-noise ratio perception. Background Art

[0002] Text-to-audio (TTA) generation technology based on diffusion model has attracted increasing attention in the field of artificial intelligence content creation, especially in applications such as movies and games that require immersive audio experience.

[0003] In the text-to-audio (TTA) generation task, the diffusion model Diffsound and the audio latent diffusion model AudioLDM can be used to effectively model data. However, the above models need to go through hundreds of sampling steps to generate high-quality samples during the generation process. This high latency limits real-time applications and cannot meet users' needs for fast feedback.

[0004] Knowledge distillation technology can build a smaller and more efficient model while maintaining model performance. Salimans et al. proposed a progressive distillation method based on a diffusion model. By reconstructing the loss weight function and The student model is trained to make its one-step predictions match the two-step predictions of the teacher model. However, this approach lacks an upper bound on the loss weight function, causing the model to be highly sensitive to small changes in predictions, especially when the signal-to-noise ratio (SNR) is high at the beginning of training.

[0005] Hang et al. improved this by introducing an upper limit on the loss weight: To avoid excessive concentration of weights in model training under high SNR conditions. However, they did not set a lower limit, resulting in greater uncertainty in the model under high noise levels and low SNR conditions in the later stages of training, reducing the effectiveness of training.

[0006] In summary, the existing progressive distillation methods based on diffusion models have not been optimized for the TTA generation task, and the existing progressive distillation methods based on diffusion models fail to take into account the uncertainty of the model under high noise levels and low signal-to-noise ratio conditions in the late stage of training, resulting in insufficient generation efficiency and quality. Summary of the invention

[0007] In order to solve the above-mentioned problems in the prior art, the present invention adopts a text-to-audio generation distillation method based on balanced signal-to-noise ratio perception, including: obtaining text, inputting the text into a trained student model for generating audio, and obtaining audio; the training process of the student model for generating audio includes:

[0008] S1. Obtain a text dataset and an audio dataset, obtain a pre-trained text encoder, an audio encoder, and a teacher model, and initialize the parameters of the student model according to the parameters of the teacher model;

[0009] S2, inputting the data in the text dataset and the audio dataset into the pre-trained text encoder and audio encoder respectively to obtain text features and audio features;

[0010] S3, inputting the text features and the audio features into the teacher model, and inputting the output result of the teacher model and the text features into the student model for generating audio;

[0011] S4. Calculate the value of the balanced signal-to-noise ratio perceptual loss function according to the output results of the teacher model and the output results of the student model, update the parameters of the student model according to the balanced signal-to-noise ratio perceptual loss function value, and determine whether the student model meets the preset training conditions. If not, return to step S3; otherwise, determine whether the preset maximum number of distillations is reached. If so, obtain a trained student model for generating audio; otherwise, set the parameters of the teacher model according to the parameters of the current student model, and return to step S3 for the next distillation.

[0012] Both the teacher model and the student model are diffusion models; the diffusion model includes a forward denoising process and a reverse denoising process, and both the forward denoising process and the reverse denoising process include multiple time steps.

[0013] The output of the student model includes the output of each time step of the reverse denoising process; the balanced signal-to-noise ratio perception loss function L θ for:

[0014]

[0015] Among them, w( ) is the loss weight function, γ t is the signal-to-noise ratio of the student model at time step t, is the output of the teacher model, is the output result of the reverse denoising process of the student model at time step t, is the noise addition result of the forward noise addition process of the student model at time step t, and τ is the text feature.

[0016] Loss weight function Among them, α t , σ t are the information level and noise level of the student model at time step t, and λ is a hyperparameter.

[0017] Preferably, the hyperparameter λ∈{5,6,7}.

[0018] The student model processes the output and text features of the teacher model, including: adding noise to the output of the teacher model to obtain the output result of the forward noise adding process of the student model; denoising the output result of the forward noise adding process of the student model according to the text features to obtain the output result of the student model; the output result of the student model includes the denoising result of each time step.

[0019] The teacher model processes text features and audio features by: adding noise to the audio features to obtain the output of the forward noise adding process of the teacher model; and denoising the output of the forward noise adding process of the teacher model according to the text features to obtain the output of the teacher model.

[0020] During the first distillation, the number of sampling steps of the forward denoising process of the student model and the teacher model is S1=N; in step S4, while setting the parameters of the teacher model according to the parameters of the current student model, the number of sampling steps S is updated. j+1 =S j / 2; where S j represents the number of sampling steps for the jth distillation.

[0021] Each time step of the reverse denoising process of the teacher model executes two DDIM steps, and each time step of the reverse denoising process of the student model executes a single DDIM step; wherein DDIM is a denoising diffusion implicit model.

[0022] Beneficial effects:

[0023] 1. The present invention uses a progressive distillation method based on a diffusion model to train a student model from text to audio, thereby improving the generation quality from text to audio; 2. The present invention combines the balanced signal-to-noise ratio perception (BSA) loss function to optimize the knowledge transfer between the teacher model and the student model used for audio generation in the progressive distillation method based on the diffusion model, so that the distillation process can maintain high stability and accuracy under different signal-to-noise ratio conditions, thereby improving the generation efficiency and quality of the student model from text to audio; 3. While combining the progressive distillation method with the balanced signal-to-noise ratio perception (BSA) loss function, the present invention reduces the sampling steps required for the student model, which can greatly improve the generation speed while maintaining the original performance, making the model more suitable for real-time or high-throughput application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A network structure diagram of a text-to-audio generation distillation method based on balanced signal-to-noise ratio perception provided by an embodiment of the present invention;

[0025] Figure 2 A flowchart of a text-to-audio generation distillation method based on balanced signal-to-noise ratio perception provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] like Figure 1 , Figure 2 As shown, the present invention adopts a text-to-audio generation distillation method based on balanced signal-to-noise ratio perception, and the figure shows the forward process, backward process and distillation process of the teacher model and the student model.

[0028] The specific steps are as follows:

[0029] Obtain text, input the text into the trained student model for generating audio, and obtain audio; the training process of the student model for generating audio includes:

[0030] S1. Get text dataset and audio datasets Get the pre-trained text encoder, audio encoder, and teacher model, and The parameters η of the student model used to generate audio are initialized Parameter θ: θ←η;

[0031] S2. Input the data in the text dataset and the audio dataset into the pre-trained text encoder and audio encoder respectively to obtain text features and audio features:

[0032] τ=f FLAN-T5 (y), z0=f AudioEncoder (x)

[0033] Among them, τ is the text feature, f FLAN-T5 () is the text encoder, y is the data of the text dataset, z0 is the audio feature, f AudioEncoder ( ) is the audio encoder, and x is the data in the audio dataset.

[0034] S3. Input text features and audio features into the teacher model Get the output of the teacher model; input the output of the teacher model and text features into the student model Get the output of the student model;

[0035] The teacher model and the student model used to generate audio are both diffusion models; the diffusion model includes a forward denoising process and a reverse denoising process; in order to accelerate the generation process and improve the generation quality, this embodiment improves the diffusion model.

[0036] The teacher model processes text features and audio features including:

[0037] In the first distillation, the sampling step number S1 of the teacher model is set to N, preferably, N is 200;

[0038] Compute time step t:

[0039] t=i / N,i∈{1,2,…,N}

[0040] From the standard normal distribution Sampled random noise ∈:

[0041]

[0042] The audio features are noised sequentially according to the time step t, and the formula is:

[0043] z t =α t z0+σ t ∈

[0044] Among them, z t is the noise result of the student model at time step t, α t , σ t are the information level and noise level at time step t, respectively;

[0045] According to the text features, the output z1 of the teacher model forward denoising process is denoised at time step t to obtain the final output Specifically include:

[0046] Each time step t includes two DDIM (Denoising Diffusion Implicit Models) steps t', t":

[0047] t'=t-0.5 / N,t"=t-1 / N

[0048] According to the two DDIM steps t' and t" at each time step t, the output of the forward denoising process of the teacher model is denoised in turn. Specifically, the denoising formula at time step t is:

[0049]

[0050] The final output of the teacher model for:

[0051]

[0052] The student model processes the output results and text features of the teacher model including:

[0053] In the first distillation, the number of sampling steps S1 of the student model is consistent with that of the teacher model, which is N;

[0054] Calculate the sampling time step t:

[0055] t=i / N,i∈{1,2,...,N}

[0056] From the standard normal distribution Sampled random noise ∈:

[0057]

[0058] According to the sampling time step t, the output of the teacher model is forward-noised in turn, and the formula is:

[0059]

[0060] in, is the noise result of the student model at time step t, α t , σ t are the information level and noise level at time step t;

[0061] De-noise the output of the forward denoising process of the student model according to the text features to obtain the final denoising result of the student model

[0062]

[0063] in, It is the denoising result of the student model at time step t-1 / N.

[0064] The teacher model starts from z at each time step t to z t-0.5 / N Then to z t-1 / N The training goal of the student model is to go directly from Predict the two-step denoising results of the teacher model This allows the student model to learn a single DDIM step from the two DDIM steps of the teacher model; where DDIM is a denoising diffusion implicit model.

[0065] S4. Calculate the loss function value according to the output results of the teacher model and the output results of the student model, update the parameters of the student model according to the loss function value, and determine whether the student model meets the preset training conditions. If not, return to step S3; otherwise, determine whether the preset maximum number of distillations is reached. If so, obtain a trained student model for generating audio; otherwise, set the parameters of the teacher model according to the parameters of the current student model: η←θ, and update the sampling step number S j+1 =S j / 2, and return to step S3 for the next distillation; wherein the preset training conditions are: the loss function value is the minimum or the maximum number of training times, S j represents the number of sampling steps for the jth distillation.

[0066] Preferably, the maximum number of distillation times in this embodiment is 4.

[0067] In this embodiment, a signal-to-noise ratio weighting term is introduced into the loss function to optimize the training process of the model under different noise levels. Specifically, the denoising result of each time step of the student model in the loss function is weighted, and the weight is adjusted according to the signal-to-noise ratio (SNR) of the denoising result.

[0068] The formula of BSA loss function is:

[0069]

[0070] Among them, w( ) is the loss weight function, γ t is the signal-to-noise ratio of the student model at time step t, and λ is a hyperparameter.

[0071] During the training process, when the SNR is high, the weight is locked to γ ​​to avoid excessive attention to low-noise information; when the SNR is low, the weight is close to 1 to prevent ignoring high-noise information. Therefore, this loss function effectively avoids the prediction bias of the model under different signal-to-noise ratio conditions and improves the quality of audio generation.

[0072] In summary, this embodiment combines the balanced signal-to-noise ratio aware (BSA) loss function to optimize the knowledge transfer between the teacher model and the student model, so that the distillation process can maintain high stability and accuracy under different signal-to-noise ratio conditions, thereby improving the generation efficiency and quality of the student model from text to audio.

[0073] As shown in Table 1, Frechet Audio Distance (FAD) and Frechet Distance (FD) are used as the main evaluation indicators of generated audio quality. FAD is an extended application of Frechet Inception Distance (FID) in the audio field, which can quantify the degree of perceptual distortion between generated audio and real audio. By comparing the generated audio with the audio features extracted by the pre-trained model, FAD provides an objective standard for measuring the quality of generated audio, which has important reference value in the evaluation of the perceptual quality of generated audio.

[0074] In the subjective evaluation part, the generated audio was evaluated subjectively in multiple dimensions based on the standard experimental settings. The evaluation content includes the overall quality (Overall Quality, OVL) of the generated audio and the relevance (Relevance, REL) to the input text. The score uses a quantitative scale of 1 to 100, and the higher the score, the higher the audio quality or relevance. By combining objective evaluation indicators with subjective scores, the quality of the generated audio is comprehensively and accurately evaluated.

[0075] Table 1 shows the data samples generated at different reasoning steps (200, 100, 50, and 25), and evaluated by objective indicators of the AudioCaps test set. The model was distilled using the BSA method, reducing the reasoning steps to 100, 50, and 25, respectively, and compared with the Salimans method and Hang's method; the confidence space was used based on the five experimental data to reduce the randomness of the data and improve the credibility of the conclusion. The experimental results show that under the same reasoning steps, the student model trained based on the BSA method outperforms the models of other methods in terms of FAD and FD indicators. Specifically, the FAD value of the 25-step student model is 1.663 and the FD is 21.856, which are lower than the models trained by other methods, indicating that the BSA method generates higher quality samples while reducing the reasoning steps. In addition, the BSA method achieves a sampling speed increase of up to 8 times compared to the undistilled model.

[0076] Table 1 Performance of teacher model and student model at different sampling steps

[0077]

[0078]

[0079] Table 2 shows the experimental results of adjusting the hyperparameter λ before the third distillation. The data shows that when the λ value is between 5 and 7, the BSA method performs better on this task.

[0080] Table 2 Performance of hyperparameter λ in different value ranges

[0081]

[0082] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A text-to-audio generation distillation method based on balanced signal-to-noise ratio perception, characterized in that: include: Get the text, input the text into the student model trained to generate audio, and get the audio; The training process of the student model for generating audio involves: S1. Obtain a text dataset and an audio dataset, obtain a pre-trained text encoder, an audio encoder, and a teacher model, and initialize the parameters of the student model according to the parameters of the teacher model; S2, inputting the data in the text dataset and the audio dataset into the pre-trained text encoder and audio encoder respectively to obtain text features and audio features; S3, inputting the text features and the audio features into the teacher model, and inputting the output result of the teacher model and the text features into the student model for generating audio; S4. Calculate the value of the balanced signal-to-noise ratio perceptual loss function according to the output results of the teacher model and the output results of the student model, update the parameters of the student model according to the balanced signal-to-noise ratio perceptual loss function value, and determine whether the student model meets the preset training conditions. If not, return to step S3; otherwise, determine whether the preset maximum number of distillations is reached. If so, obtain a trained student model for generating audio; otherwise, set the parameters of the teacher model according to the parameters of the current student model, and return to step S3 for the next distillation.

2. The text-to-audio generation distillation method based on balanced signal-to-noise ratio perception according to claim 1, characterized in that: Both the teacher model and the student model are diffusion models; the diffusion model includes a forward denoising process and a reverse denoising process, and both the forward denoising process and the reverse denoising process include multiple time steps.

3. The text-to-audio generation distillation method based on balanced signal-to-noise ratio perception according to claim 2, characterized in that: The output of the student model includes the output of each time step of the reverse denoising process; the balanced signal-to-noise ratio perception loss function L θ for: Among them, w() is the loss weight function, γ t is the signal-to-noise ratio of the student model at time step t, is the output of the teacher model, is the output result of the reverse denoising process of the student model at time step t, is the noise addition result of the forward noise addition process of the student model at time step t, and τ is the text feature.

4. The text-to-audio generation distillation method based on balanced signal-to-noise ratio perception according to claim 3 is characterized in that: Loss weight function Among them, α t , σ t are the information level and noise level of the student model at time step t, and λ is a hyperparameter.

5. The text-to-audio generation distillation method based on balanced signal-to-noise ratio perception according to claim 4, characterized in that: Hyperparameter λ∈{5,6,7}.

6. The text-to-audio generation distillation method based on balanced signal-to-noise ratio perception according to claim 2, characterized in that: The student model processes the output and text features of the teacher model, including: adding noise to the output of the teacher model to obtain the output result of the forward noise adding process of the student model; denoising the output result of the forward noise adding process of the student model according to the text features to obtain the output result of the student model; the output result of the student model includes the denoising result of each time step.

7. The text-to-audio generation distillation method based on balanced signal-to-noise ratio perception according to claim 2, characterized in that: The teacher model processes text features and audio features by: adding noise to the audio features to obtain the output of the forward noise adding process of the teacher model; and denoising the output of the forward noise adding process of the teacher model according to the text features to obtain the output of the teacher model.

8. The text-to-audio generation distillation method based on balanced signal-to-noise ratio perception according to claim 2, characterized in that: During the first distillation, the sampling steps of the student model and the teacher model are S1=N; in step S4, the parameters of the teacher model are set according to the parameters of the current student model, and the sampling steps S are updated. j+1 =S j / 2; where S j represents the number of sampling steps for the jth distillation.

9. The text-to-audio generation distillation method based on balanced signal-to-noise ratio perception according to claim 2, characterized in that: Each time step of the reverse denoising process of the teacher model executes two DDIM steps, and each time step of the reverse denoising process of the student model executes a single DDIM step; wherein DDIM is a denoising diffusion implicit model.

Images