Fast text guidance sound effect generation method and system based on segmented rectification

Abstract

The invention discloses a rapid text guidance sound effect generation method and system based on segmented rectification, and belongs to the field of sound effect generation. The method comprises the following steps: acquiring sound effect data with description text annotations, pre-training a diffusion model as a teacher diffusion model, training a student diffusion model by adopting distillation learning, taking a differential equation trajectory of the student diffusion model as a segmented linear trajectory, and calculating by adopting a linear interpolation mode to obtain noise-added Mel spectrogram features corresponding to sampling time steps; the student diffusion model is guided by text features of a description text, and noise is predicted for noise-added Mel spectrogram features corresponding to sampling time steps; in the sound effect generation stage, a user provides a description text and initializes noise, and the student diffusion model operates an inverse diffusion denoising process segment by segment and generates a final sound effect. According to the method, the segmented rectification technology is introduced into the sound effect generation diffusion model, so that rapid text-guided sound effect generation is realized, and the sound effect quality generated during low step number generation of the diffusion model is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of sound effect generation, and in particular to a method and system for generating fast text-guided sound effects based on segmented rectification. Background Art

[0002] Nowadays, automatic sound effect generation has attracted widespread attention and has become an important field of artificial intelligence. This field aims to allow users to automatically generate sound effects by inputting descriptive text about a sound effect. It plays an important role in the development of movie dubbing, game dubbing, short video production, and virtual reality technology.

[0003] Existing technologies can be divided into two categories: 1) sound effect generation models based on autoregressive models, and 2) sound effect generation models based on diffusion models.

[0004] The autoregressive-based sound effect generation model encodes the sound effect waveform into a discrete audio latent variable sequence using a vector quantized variational autoencoder, and generates discrete latent variable labels one by one, achieving good performance. However, due to the limitations of its model structure, the model needs to be run once for each discrete latent variable. For a discrete latent variable sequence of length T, the model needs to be run T times, which consumes a lot of time and is difficult to deploy in scenarios with high performance requirements. Although the prior art uses a vector quantized variational autoencoder to compress waveform data into discrete latent codes to reduce the length of the audio data sequence, thereby reducing the number of times the autoregressive model needs to be run, the compression capability of the data length is limited, and too high a compression ratio will also correspond to lower reconstruction quality and reduce the quality of the audio generated by the autoregressive model. The existing sound effect generation model based on the diffusion model also needs to run the model multiple times for multi-step denoising to achieve higher sound effect generation quality. Although the existing technology can take a subsequence of the time step sequence of the diffusion model to achieve generation with a lower number of steps, the inverse diffusion step used by the diffusion model is an ordinary differential equation solver (ODESolver) obtained based on the diffusion function. Due to the limitations of the mathematical form of the inverse solver, the trajectory of the ordinary differential equation is a curved trajectory. When the number of sampling times is too small, it is inevitable to further expand the error, and the number of steps can be reduced to a limited extent, which makes the generation speed of the sound effect generation model slow, and also limits the deployment of the sound effect generation model to scenarios with high performance requirements. Summary of the invention

[0005] In order to overcome the problem of slow generation speed of existing sound effect generation models, the present invention provides a fast text-guided sound effect generation method and system based on piecewise rectification, which uses a pre-trained teacher diffusion model to distill a student diffusion model with the same structure as the teacher diffusion model with a piecewise rectification objective. The student diffusion model can generate high-quality sound effects with a far lower inverse diffusion step than the teacher diffusion model, so as to achieve fast text-guided sound effect generation.

[0006] In order to achieve the above object, the specific technical solution adopted by the present invention is:

[0007] In a first aspect, the present invention proposes a method for generating fast text-guided sound effects based on segmented rectification, comprising:

[0008] A training set consisting of sound effect waveform data and its description text is obtained, and a diffusion model is pre-trained as a teacher diffusion model; during pre-training, the teacher diffusion model is guided by the text features of the description text to gradually predict noise based on the noisy Mel-spectrogram features of the sound effect waveform data;

[0009] Using the teacher diffusion model to train a student diffusion model using distillation learning, the time step is divided into a number of time periods based on a predefined number of segments, and the differential equation trajectory of the student diffusion model is a straight line trajectory in each segment;

[0010] In each training batch of the distillation learning phase, randomly sample time steps in a random time period, calculate the noisy Mel-spectrogram features corresponding to the first and last time steps in the selected random time period of the teacher diffusion model, and use linear interpolation to calculate the noisy Mel-spectrogram features corresponding to the sampling time step; the student diffusion model is guided by the text features describing the text, predicts the noise of the noisy Mel-spectrogram features corresponding to the sampling time step, and calculates the loss with the inter-segment denoising noise of the teacher diffusion model to update the student diffusion model;

[0011] In the sound effect generation stage, the user provides a description text and initializes the noise. The student diffusion model is guided by the text features of the description text, runs the inverse diffusion denoising process section by section, and generates the final sound effect.

[0012] Furthermore, the teacher diffusion model and the student diffusion model are diffusion models with the same structure.

[0013] Furthermore, the teacher diffusion model and the student diffusion model include:

[0014] The sinusoidal coding layer is used to perform sinusoidal coding on the time step to obtain the time step characteristics of the diffusion model;

[0015] The first fully connected layer is used to project the diffusion model time step features to generate time step embedding features;

[0016] The second fully connected layer is used to project text features to generate text embedding features;

[0017] A one-dimensional convolutional layer for projecting the noisy Mel-spectrogram features to generate noisy Mel-spectrogram embedding features;

[0018] N stacked Transformer blocks, which are used to extract encoding features from the concatenation of time-step embedding features, text embedding features, and noisy Mel-spectrogram embedding features;

[0019] Group normalization and 1D convolutional layers, which take the encoded features as input and generate output sequences;

[0020] The prediction noise output layer is used to cut out the part corresponding to the sound effect waveform data from the output sequence as the prediction noise.

[0021] Furthermore, during the pre-training phase of the teacher diffusion model, the diffusion model is updated with the loss between the predicted noise and the actual noise of the diffusion model.

[0022] Furthermore, the distillation learning phase of the student diffusion model includes:

[0023] Initialize the student diffusion model parameters using the teacher diffusion model parameters and divide the total time step T into several time periods;

[0024] Sample a training batch of data from a dataset D, where the dataset D consists of the Mel-spectrogram features corresponding to the training set and text features Composition; perform the following steps a to d in each training batch until the student diffusion model converges:

[0025] Step a: Randomly select one of the time periods , and randomly sample time steps within the selected time period , calculate the teacher diffusion model and the first and last time steps of the selected time period , The corresponding time step , ;

[0026] Step b, based on Mel-spectrogram features and random noise to run the diffusion and noise process, and obtain the teacher diffusion model at time step Noise-added Mel-spectrogram features ; Based on the noisy Mel spectrum features and text features Step by step, run the inverse diffusion de-noising process of the teacher diffusion model until the teacher diffusion model is obtained at time step Noise-added Mel-spectrogram features ;

[0027] Step c: use linear interpolation to calculate the sampling time step Corresponding noisy Mel-spectrogram features :

[0028] Step d, student diffusion model based on text features To guide, the sampling time step Corresponding noisy Mel-spectrogram features Predict the noise and calculate the loss function to update the Student Diffusion Model.

[0029] Furthermore, in step a, the time step , The calculation formula is as follows:

[0030] ;

[0031] ;

[0032] in, Indicates rounding down.

[0033] Furthermore, in step c, the formula of the linear interpolation method is as follows:

[0034] .

[0035] Furthermore, in step d, the loss function is calculated as follows:

[0036] ;

[0037] in, represents the student diffusion model loss, represents the parameters of the Student diffusion model, represents the student diffusion model, represents the mean coefficient of the Gaussian distribution, represents the variance coefficient of the Gaussian distribution, represents the norm, is the inter-segment denoising noise of the teacher diffusion model.

[0038] Furthermore, the sound effect generation stage of the student diffusion model includes:

[0039] Extract text features of the description text provided by the user, and,randomly sample Gaussian noise;

[0040] The total time step is divided into several time periods, and each time period is traversed in order from the last time period to the first time period. The traversal process is as follows: guided by the text features describing the text, the inverse diffusion denoising process of the student diffusion model is gradually run in each time period, and finally a noise-free Mel-spectrogram feature is generated;

[0041] The pre-trained decoder is used to decode the noise-free Mel-spectrogram features to obtain a Mel-spectrogram, and then the pre-trained vocoder is used to convert the Mel-spectrogram into an audio waveform.

[0042] In a second aspect, the present invention provides a fast text guidance sound effect generation system based on segmented rectification, which is used to implement the above-mentioned fast text guidance sound effect generation method based on segmented rectification.

[0043] The beneficial effects of the present invention are:

[0044] The present invention first pre-trains a teacher diffusion model to achieve the purpose of gradually predicting noise based on the noisy Mel spectrum features of sound effect waveform data, and then uses the teacher diffusion model to train a student diffusion model by distillation learning. The student diffusion model divides the trajectory into multiple segments based on the curved ordinary differential equation trajectory of the classroom diffusion model, and performs distillation learning with a linear rectification target in each segment. The noisy Mel spectrum features corresponding to the sampling time step in the student diffusion model are calculated by linear interpolation, so the student diffusion model can learn the straight ordinary differential equation trajectory in the segment; since the expected ordinary differential equation trajectory in each segment becomes a straight line trajectory, the student diffusion model can use a larger sampling step size without causing excessive errors, and can generate sound effects of the same quality with a generation step number far lower than that of the teacher diffusion model, overcoming the problem of slow generation speed of the existing sound effect generation model and greatly improving the quality of sound effects generated by the diffusion model with a low number of steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a schematic diagram of a method for generating fast text-guided sound effects based on segmented rectification according to an embodiment of the present invention.

[0046] Figure 2 It is a schematic diagram of the diffusion process and the reverse diffusion process shown in an embodiment of the present invention.

[0047] Figure 3 It is a structural diagram of the diffusion model. DETAILED DESCRIPTION

[0048] The present invention is further described and illustrated below in conjunction with specific embodiments. The embodiments are merely exemplary of the present disclosure and do not define the scope of limitation. The technical features of each embodiment of the present invention may be combined accordingly without conflicting with each other.

[0049] The accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0050] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0051] Figure 1 The figure shows a schematic diagram of a fast text-guided sound effect generation method based on segmented rectification of the present invention. In the training stage, a sound effect and its description text are input to train the diffusion model to realize text-guided sound effect generation. The diffusion model obtained by the above training is used as the teacher diffusion model, and the student diffusion model with the same structure as the teacher diffusion model is distilled with the segmented rectification objective. In the inference stage, a description text of the sound effect is input and Gaussian sampling noise is added. The student diffusion model is segmented and gradually denoised to generate the final sound effect. The student diffusion model can generate high-quality sound effects with a much lower inverse diffusion step than the teacher diffusion model.

[0052] Specifically, the present invention proposes a method for generating a fast text-guided sound effect based on segmented rectification, which mainly includes the following steps:

[0053] Step 1, obtain sound effect waveform data and description text data of the sound effect waveform data as a training set; pre-process the sound effect waveform data to obtain sound effect Mel-spectrogram data.

[0054] Step 2: extracting mel-spectrogram features of the sound effect mel-spectrogram data, and extracting text features of the description text.

[0055] Step 3: randomly sample time step t, and add noise to the Mel-spectrogram feature using a diffusion process to obtain a noisy Mel-spectrogram feature.

[0056] Step 4, taking the sampling time step, text features and noisy Mel-spectrogram features as inputs of the diffusion model, the diffusion model predicts the noise of the sampling time step, taking the mean square error loss of the actual noise added to the Mel-spectrogram features at the sampling time step and the predicted noise of the diffusion model as the loss function, and taking the trained diffusion model as the teacher diffusion model;

[0057] Step 5, define a student diffusion model with the same structure as the teacher diffusion model, initialize the student diffusion model with the teacher diffusion model parameters, and train the student diffusion model in a piecewise rectification manner using the training set and the teacher diffusion model; the student diffusion model can generate sound effects of the same quality with a much lower number of generation steps than the teacher diffusion model, thereby achieving fast text-guided sound effect generation.

[0058] In step 6, the user provides a description text and extracts text features. The student diffusion model samples noise from a Gaussian distribution, and performs an inverse diffusion process on the sampling result with the text feature input as a condition to generate a Mel-spectrogram feature. The Mel-spectrogram feature is decoded to obtain a Mel-spectrogram, which is then converted into an audio waveform through a pre-trained vocoder to obtain the final sound effect.

[0059] In the above step 1, the process of preprocessing the sound effect waveform data is: resampling the sound effect waveform data, and converting the sound effect waveform data into sound effect Mel spectrum data through short-time Fourier transform and Mel filter. In this embodiment, for the sound effect waveform-description text data pair (a, y), is an audio waveform. For an audio waveform resampled to 16000 Hz, if the audio waveform duration is n seconds, then . Indicates the description text corresponding to the sound effect waveform. The sound effect waveform data is converted into the sound effect Mel spectrum data through short-time Fourier transform and Mel filter. ,in is the number of Mel barrels, is the number of mel frames, Much smaller than .

[0060] In the above step 2, an optional implementation process for obtaining the Mel-spectrogram features and text features is as follows:

[0061] Mel-spectrogram for sound effects , use the pre-trained Mel-spectrogram encoder to encode the Mel-spectrogram features:

[0062]

[0063] Where E is the pre-trained Mel-spectrogram encoder, , are the downsampling rates of the number of Mel buckets and the number of Mel frames, respectively. is a Mel-spectrogram feature. In this embodiment, the Mel-spectrogram encoder uses the encoder part of the variational autoencoder. The structure and pre-training process of the variational autoencoder are common knowledge in the art and will not be described here.

[0064] For description text , using the pre-trained text encoder Convert to text features In this embodiment, the pre-trained text encoder is implemented using the existing Flan-T5-Large model, which will not be described in detail here.

[0065] In step 3 above, if Figure 2 As shown in the figure, the process of adding noise to the Mel-spectrogram feature using the diffusion process is as follows:

[0066] Taking the diffusion process as an example, it can be expressed as:

[0067]

[0068] According to the properties of Gaussian distribution, it can be deduced that:

[0069]

[0070] Assume there are T time steps, sample time step t from a uniform distribution {1,2,…,T}, and use the predefined time step parameter , for Mel-spectrogram features according to Noise is added in the form of represents the noisy Mel-spectrogram feature, The sampling is from the mean of 0 and the variance of Gaussian noise distribution, the sampling noise and Mel frequency spectrum features Same shape, is the identity matrix; represents the product of the parameters of the first t time steps, Represents the parameters of the i-th time step.

[0071] In step 4 above, the reverse diffusion process adopts Figure 3 The diffusion model shown is implemented by Gradually restore to normal The diffusion model includes a sinusoidal coding layer, a first fully connected layer, a second fully connected layer and a one-dimensional convolutional layer; the sinusoidal coding layer is used to perform sinusoidal coding on the sampling time step t to obtain the diffusion model time step feature; the diffusion model time step feature, text feature and noisy Mel spectrum map feature are respectively projected to the feature space of the diffusion model by using the learnable first fully connected layer, the second fully connected layer and the one-dimensional convolutional layer to obtain the time step embedding feature, the text embedding feature and the noisy Mel spectrum map embedding feature, the three embedding features are connected and input into the Transformer block to extract features, the results after superimposing N Transformer blocks are then obtained by group normalization and one-dimensional convolution to obtain the output sequence, and the part with the same length as the number of Mel frames is cropped from the output sequence as the prediction noise.

[0072] In a specific implementation of the present invention, an optional implementation process of using a diffusion model to predict the noise at a sampling time step t is as follows:

[0073] 4.1) First, encode the time step embedding features, text embedding features, and noisy Mel spectrogram embedding features:

[0074]

[0075]

[0076]

[0077]

[0078] in, represents two learnable first and second fully connected layers, represents sinusoidal coding, represents a learnable one-dimensional convolutional layer, Indicates feature connection; represents the randomly sampled time step, represents the text features extracted by the text encoder, Represents the noisy Mel-spectrogram feature; , , They represent the time step embedding features, text embedding features and noisy Mel-spectrogram embedding features respectively; h represents the total feature after connection, and d is the dimension of the feature space of the Transformer block in the diffusion model.

[0079] 4.2) The total features after connection The input Transformer block is processed by N Transformer blocks in sequence, and the generated results are then group normalized and one-dimensional convolution to obtain the output sequence.

[0080] 4.3) Extract prediction noise from the output sequence.

[0081] During the operation of the diffusion model, the channel dimension of the output sequence is the same as the Mel-spectrogram feature dimension, and the time dimension length of the output result is the same as the total feature time dimension after concatenation ( ). In order to make the time dimension length the same as the time dimension length of the Mel-spectrogram feature, the length of the rear of the cropped output is The part of is used as the final prediction noise, that is, .

[0082] 4.4) The mean square error loss of the actual noise added to the Mel-spectrogram feature at time step t and the predicted noise of the diffusion model is used as the loss function. The loss function is expressed as follows:

[0083]

[0084] in, represents the diffusion model;

[0085] The parameters obtained by training are The diffusion model of is used as the teacher diffusion model.

[0086] In step 5 above, an optional implementation of training the student diffusion model is as follows:

[0087] 5.1) Define the following parameters:

[0088] Define the data set D, which consists of the Mel-spectrogram features corresponding to the training set data and text features composition.

[0089] The teacher diffusion model is denoted as , the student diffusion model is denoted as , the time step parameter of the teacher diffusion model is recorded as , the reverse diffusion sampling step of the teacher diffusion model is recorded as The number of segments is recorded as ,create Time periods:

[0090]

[0091] in, =0, .

[0092] 5.2) Initialize the parameters of the student diffusion model:

[0093] Using teacher diffusion model parameters Initialize the parameters of the Student Diffusion Model : .

[0094] 5.3) From the dataset Sample a mini-batch of text features and Mel-spectrogram features ,in Text features representing the descriptive text of the sound effect, Represents a mel-spectrogram feature.

[0095] 5.4) Randomly sample a number k from {1,…,K}, and select Randomly sample time steps from a uniform distribution , calculate the teacher diffusion model and , The corresponding time step is:

[0096]

[0097]

[0098] in, is the total number of time steps of the teacher diffusion model, Indicates rounding down.

[0099] 5.5) In the time period Medium Calculation and , obtained by interpolation ;

[0100] Specifically, first run the diffusion noise process to get the corresponding Step-by-step noisy Mel-spectrogram features:

[0101]

[0102] Afterwards, Based on, run the inverse diffusion step of the teacher diffusion model until we get , the process is:

[0103] remember , , updated at each step:

[0104]

[0105]

[0106] Until ,at this time That is .

[0107] get postscript:

[0108]

[0109]

[0110] in, represents the mean coefficient of the Gaussian distribution, represents the variance coefficient of Gaussian distribution;

[0111] Finally, the problem to be solved is the teacher diffusion model in Steps and The linear interpolation of the step-noise feature is obtained:

[0112]

[0113] 5.6) The loss function for calculating the student diffusion model is:

[0114]

[0115] Compute the gradient of the loss function on this batch and use the optimizer to optimize the parameters of the Student Diffusion Model to update.

[0116] 5.7) Return to step 5.3) and continue to traverse the data set To update the student diffusion model parameters , until the loss function converges; since the pre-trained teacher diffusion model parameters are used to initialize the student diffusion model parameters, the loss function of the student diffusion model can converge quickly.

[0117] Since the input of the student diffusion model is through and Therefore, the student diffusion model can learn the ordinary differential equation trajectory of the straight line in the segment, and can use a larger step size than the teacher diffusion model for sampling in each segment without large numerical errors, thereby greatly reducing the number of sampling steps and increasing the speed of sound effect generation. The student diffusion model can generate sound effects of the same quality with a much lower number of generation steps than the teacher diffusion model, thereby realizing a fast text-guided sound effect generation system.

[0118] In step 6 above, an optional implementation process of generating sound effects in a text-guided manner using the student diffusion model is as follows:

[0119] 6.1) Define the following parameters:

[0120] The student diffusion model is denoted as , the number of segments is K, the number of sampling steps in each segment is m, and the sound effect description text is ; Sample noise from Gaussian noise .

[0121] 6.2) Use the Student Diffusion Model to implement the following inverse diffusion process:

[0122] set up , , ,in represents the generation stride of the student diffusion model;

[0123] make , , segment location and The corresponding teacher diffusion model time step multiplication parameter is and , then the Gaussian distribution mean coefficient corresponding to the teacher diffusion model DDIM sampling can be calculated and the coefficient of variance :

[0124]

[0125]

[0126] when When, according to the DDIM sampling method, record the random sampling In the paragraph The Gaussian distribution mean coefficient after linear interpolation of the inner position is:

[0127]

[0128] Note the Gaussian distribution variance coefficient after linear interpolation for:

[0129]

[0130] The end position of the paragraph predicted by the student model for:

[0131]

[0132] Then the direction vector of the Euler solver at each step of the Student diffusion model is for:

[0133]

[0134] So at each step the Euler solver is run to update and t:

[0135]

[0136]

[0137] until until; when When , the student diffusion model runs to the end of the segment, and the segment is updated:

[0138]

[0139]

[0140] Start running the next segment of sampling.

[0141] Until So far, the student diffusion model has successfully denoised the Gaussian noise. is the clean Mel-spectrogram feature, which is recorded as .

[0142] 6.3) Using the decoder to decode the Mel-spectrogram features The mel-spectrogram is decoded and then converted into an audio waveform by a pre-trained vocoder. In this embodiment, the mel-spectrogram decoder uses the decoder part of the variational autoencoder. The structure and pre-training process of the variational autoencoder are common knowledge in the art and will not be described here.

[0143] The above method is applied to the following embodiments to demonstrate the technical effects of the present invention, and the specific steps in the embodiments are not repeated here.

[0144] The present invention evaluates the sound effect generation index on the test set of the sound effect-description text dataset AudioCaps. The generation indexes of the TANGO sound effect generation model, the teacher diffusion model and the student diffusion model are compared. The display is shown in Table 1. IS and FAD are sound effect quality evaluation indicators, and KL and CLAP are speech-text alignment indicators. The larger the IS and CLAP, the better, and the smaller the FAD and KL, the better.

[0145] Table 1: Test results of the AudioCaps sound effect dataset of the present invention

[0146]

[0147] As can be seen from Table 1, when generating with a low number of steps of 4, the student diffusion model has a significant advantage over the teacher diffusion model and the TANGO sound effect generation model; after increasing the number of steps, the three models gradually become equal. The method proposed in the present invention can generate sound effects of comparable quality with a number of steps far lower than that of the comparison model, thus achieving fast text-guided sound effect generation.

[0148] Based on the same inventive concept, a fast text-guided sound effect generation system based on segmented rectification is also provided in the present embodiment, and the system is used to implement the above-mentioned embodiment. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware of predetermined functions. Although the system described in the following embodiments is preferably implemented in software, it is also possible to implement hardware, or a combination of software and hardware.

[0149] In this embodiment, a fast text-guided sound effect generation system based on segmented rectification includes:

[0150] A teacher diffusion model pre-training module is used to obtain a training set consisting of sound effect waveform data and its description text, and pre-train a diffusion model as a teacher diffusion model; during pre-training, the teacher diffusion model is guided by the text features of the description text to gradually predict the noise based on the noisy Mel-spectrogram features of the sound effect waveform data;

[0151] A distillation learning module based on a linear rectification target is used to train a student diffusion model using distillation learning using a teacher diffusion model, wherein the differential equation trajectory of the student diffusion model is a straight line trajectory, and the time step is divided into several time periods based on the straight line trajectory; in each training batch of the distillation learning stage, the time step is randomly sampled in a random time period, and the noisy Mel-spectrogram features corresponding to the first and last time steps of the teacher diffusion model in the selected random time period are calculated, and the noisy Mel-spectrogram features corresponding to the sampled time steps are calculated using a linear interpolation method; the student diffusion model is guided by the text features describing the text, and the noise of the noisy Mel-spectrogram features corresponding to the sampling time steps is predicted, and the loss is calculated with the inter-segment denoising noise of the teacher diffusion model to update the student diffusion model;

[0152] The sound effect generation module based on the student diffusion model initializes the noise, uses the student diffusion model to guide the text features of the description text provided by the user, runs the inverse diffusion denoising process section by section and generates the final sound effect.

[0153] In a specific implementation of the present invention, it may also include: a text encoder, which is used to convert the description text into text features;

[0154] Mel-spectrogram variational autoencoder, whose encoder part is used to extract the Mel-spectrogram features of the sound effect waveform data, and the decoder part is used to reconstruct the Mel-spectrogram features into a Mel-spectrogram;

[0155] A vocoder is used to convert the mel-spectrogram into an audio waveform.

[0156] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0157] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0158] The above examples are only specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many variations are possible. All variations that can be directly derived or associated with the contents disclosed by a person skilled in the art should be considered as the protection scope of the present invention.

Claims

1. A method for generating fast text-guided sound effects based on segmented rectification, characterized in that: include: A training set consisting of sound effect waveform data and its description text is obtained, and a diffusion model is pre-trained as a teacher diffusion model; during pre-training, the teacher diffusion model is guided by the text features of the description text to gradually predict noise based on the noisy Mel-spectrogram features of the sound effect waveform data; Using the teacher diffusion model to train a student diffusion model using distillation learning, the time step is divided into a number of time periods based on a predefined number of segments, and the differential equation trajectory of the student diffusion model is a straight line trajectory in each segment; In each training batch of the distillation learning phase, randomly sample time steps in a random time period, calculate the noisy Mel-spectrogram features corresponding to the first and last time steps in the selected random time period of the teacher diffusion model, and use linear interpolation to calculate the noisy Mel-spectrogram features corresponding to the sampling time step; the student diffusion model is guided by the text features describing the text, predicts the noise of the noisy Mel-spectrogram features corresponding to the sampling time step, and calculates the loss with the inter-segment denoising noise of the teacher diffusion model to update the student diffusion model; In the sound effect generation stage, the user provides a description text and initializes the noise. The student diffusion model is guided by the text features of the description text, runs the inverse diffusion denoising process section by section, and generates the final sound effect.

2. The method for generating fast text-guided sound effects based on segmented rectification according to claim 1, characterized in that: The teacher diffusion model and the student diffusion model are diffusion models with the same structure.

3. The method for generating fast text-guided sound effects based on segmented rectification according to claim 2, characterized in that: The teacher diffusion model and the student diffusion model include: The sinusoidal coding layer is used to perform sinusoidal coding on the time step to obtain the time step characteristics of the diffusion model; The first fully connected layer is used to project the diffusion model time step features to generate time step embedding features; The second fully connected layer is used to project text features to generate text embedding features; A one-dimensional convolutional layer for projecting the noisy Mel-spectrogram features to generate noisy Mel-spectrogram embedding features; N stacked Transformer blocks, which are used to extract encoding features from the concatenation of time-step embedding features, text embedding features, and noisy Mel-spectrogram embedding features; Group normalization and 1D convolutional layers, which take the encoded features as input and generate output sequences; The prediction noise output layer is used to cut out the part corresponding to the sound effect waveform data from the output sequence as the prediction noise.

4. The method for generating fast text-guided sound effects based on segmented rectification according to claim 1 or 3, characterized in that: During the pre-training phase of the teacher diffusion model, the diffusion model is updated with the loss between the predicted noise and the actual noise of the diffusion model.

5. The method for generating fast text-guided sound effects based on segmented rectification according to claim 3, characterized in that: The distillation learning phase of the student diffusion model includes: Initialize the student diffusion model parameters using the teacher diffusion model parameters and divide the total time step T into several time periods; Sample a training batch of data from a dataset D, where the dataset D consists of the Mel-spectrogram features corresponding to the training set and text features Composition; perform the following steps a to d in each training batch until the student diffusion model converges: Step a: Randomly select a time period , and randomly sample time steps within the selected time period , calculate the teacher diffusion model and the first and last time steps of the selected time period , The corresponding time step , ; Step b, based on Mel-spectrogram features and random noise to run the diffusion and noise process, and obtain the teacher diffusion model at time step Noise-added Mel-spectrogram features ; Based on the noisy Mel spectrum features and text features Step by step, run the inverse diffusion de-noising process of the teacher diffusion model until the teacher diffusion model is obtained at time step Noise-added Mel-spectrogram features ; Step c: use linear interpolation to calculate the sampling time step Corresponding noisy Mel-spectrogram features : Step d, student diffusion model based on text features To guide, the sampling time step Corresponding noisy Mel-spectrogram features Predict the noise and calculate the loss function to update the Student Diffusion Model.

6. The method for generating fast text-guided sound effects based on segmented rectification according to claim 5, characterized in that: In step a, the time step , The calculation formula is as follows: ； ； in, Indicates rounding down.

7. The method for generating fast text-guided sound effects based on segmented rectification according to claim 5, characterized in that: In step c, the formula for linear interpolation is as follows: 。 8. The method for generating fast text-guided sound effects based on segmented rectification according to claim 5, characterized in that: In step a and step d, the loss function is calculated as follows: ； in, represents the student diffusion model loss, represents the parameters of the Student diffusion model, represents the student diffusion model, represents the mean coefficient of the Gaussian distribution, represents the variance coefficient of the Gaussian distribution, represents the norm, is the inter-segment denoising noise of the teacher diffusion model.

9. The method for generating fast text-guided sound effects based on segmented rectification according to claim 3, characterized in that: The sound effect generation stage of the student diffusion model includes: Extract text features of the description text provided by the user, and,randomly sample Gaussian noise; The total time step is divided into several time periods, and each time period is traversed in order from the last time period to the first time period. The traversal process is as follows: guided by the text features describing the text, the inverse diffusion denoising process of the student diffusion model is gradually run in each time period, and finally a noise-free Mel-spectrogram feature is generated; The pre-trained decoder is used to decode the noise-free Mel-spectrogram features to obtain a Mel-spectrogram, and then the pre-trained vocoder is used to convert the Mel-spectrogram into an audio waveform.

10. A fast text-guided sound effect generation system based on segmented rectification, used to implement the method of claim 1; characterized in that: The system comprises: A teacher diffusion model pre-training module is used to obtain a training set consisting of sound effect waveform data and its description text, and pre-train a diffusion model as a teacher diffusion model; during pre-training, the teacher diffusion model is guided by the text features of the description text to gradually predict the noise based on the noisy Mel-spectrogram features of the sound effect waveform data; A distillation learning module based on a linear rectification target is used to train a student diffusion model using distillation learning using a teacher diffusion model, and divide the time step into several time periods based on a predefined number of segments, and the differential equation trajectory of the student diffusion model is a straight line trajectory in each segment; in each training batch of the distillation learning stage, randomly sample time steps in a random time period, calculate the noisy Mel spectrum features corresponding to the first and last time steps of the teacher diffusion model in the selected random time period, and use linear interpolation to calculate the noisy Mel spectrum features corresponding to the sampled time steps; the student diffusion model is guided by the text features describing the text, predicts the noise of the noisy Mel spectrum features corresponding to the sampling time steps, and calculates the loss with the inter-segment denoising noise of the teacher diffusion model to update the student diffusion model; The sound effect generation module based on the student diffusion model initializes the noise, uses the student diffusion model to guide the text features of the description text provided by the user, runs the inverse diffusion denoising process section by section and generates the final sound effect.

Images