A speech enhancement method based on Actor-Critic algorithm and diffusion model

Abstract

This application discloses a speech enhancement method based on the Actor-Critic algorithm and diffusion model, which relates to the field of speech enhancement. The method includes: using the NCSN++ model used for diffusion as an Actor model, training the model using a speech enhancement dataset and the Actor-Critic algorithm, using a speech quality assessment model as feedback given by the environment, so that the model training is aimed at improving speech quality, giving a reward of 1 for each step in the diffusion process, using the trained model as the speech enhancement model to be fine-tuned, and using this model as the Actor model. The model is trained using an unsupervised speech dataset and the Actor-Critic algorithm to fine-tune the model, thereby obtaining a final speech enhancement model. This application can achieve excellent speech enhancement effects in different scenarios, and can also quickly obtain clean speech with good quality.

Description

Technical Field

[0001] The present application relates to the field of speech enhancement technology, and in particular to a speech enhancement method based on an Actor-Critic algorithm and a diffusion model. Background Art

[0002] Currently, most speech enhancement research focuses on processing the speech signal of a single speaker, typically pure speech recorded in a laboratory environment, and tends to be more theoretical. However, in practical applications, speech signals are often affected by various noise and interference, such as background noise, echo, and reverberation, which places demands on the generalization performance of existing speech enhancement models.

[0003] Speech enhancement technology faces complex application scenarios, and traditional speech enhancement methods exhibit significant limitations in these scenarios. Whenever the application domain changes, speech enhancement models need to be fine-tuned to adapt to the new environment. In recent years, speech enhancement methods based on diffusion models have addressed this issue to some extent. However, their training still relies on a large number of clean and noisy corpus pairs and a specific SDE equation, which limits the generalization performance of diffusion models. Furthermore, these speech enhancement models suffer from long inference times.

[0004] In summary, speech enhancement based on diffusion models suffers from limited generalization performance and long inference time. Therefore, existing speech enhancement methods based on diffusion models cannot achieve good speech enhancement effects in different scenarios, nor can they quickly obtain clean speech of good quality. Summary of the Invention

[0005] The purpose of this application is to provide a speech enhancement method based on the Actor-Critic algorithm and the diffusion model, which can achieve a good speech enhancement effect in different scenarios and can also obtain clean speech with good quality relatively quickly.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a speech enhancement method based on an Actor-Critic algorithm and a diffusion model, the speech enhancement method based on the Actor-Critic algorithm and the diffusion model comprising:

[0008] Acquire a speech enhancement dataset, a speech evaluation dataset, and an unsupervised speech dataset; the speech enhancement dataset includes the VoiceBank-DEMAND dataset; the speech evaluation dataset includes the NISQA, PSTN, and Tencent datasets; and the unsupervised speech dataset includes different speech sounds;

[0009] Using the speech evaluation dataset to train the NISQA model, and using the trained NISQA model as a speech quality evaluation model;

[0010] The NCSN++ model used for diffusion is used as the Actor model. A multi-layer Transformer stack followed by an adaptive pooling layer is used as the Critic model. The NCSN++ model is trained using the speech enhancement dataset and the Actor-Critic algorithm. The speech quality assessment model is used as feedback from the environment to train the NCSN++ model with the goal of improving speech quality. A reward of -1 is given for each step in the diffusion process. The trained NCSN++ model is used as the speech enhancement model to be fine-tuned.

[0011] The speech enhancement model to be fine-tuned is used as the Actor model, the Critic model parameters are initialized, the speech enhancement model to be fine-tuned is trained using the unsupervised speech dataset and the Actor-Critic algorithm, the speech quality assessment model is used as feedback given by the environment, and the speech enhancement model to be fine-tuned is guided to improve speech quality as the goal, a reward of -1 is given for each step in the diffusion process, and the trained speech enhancement model to be fine-tuned is used as the final speech enhancement model;

[0012] The final speech enhancement model is used to perform speech enhancement on the speech to be enhanced to obtain clean speech.

[0013] Optionally, using the speech evaluation dataset to train a NISQA model, and using the trained NISQA model as a speech quality evaluation model, specifically includes:

[0014] The speech evaluation dataset is used to train the NISQA model. The input of the NISQA model is speech data, the output of the NISQA model is a speech quality evaluation score, and the loss function is the square difference between the score predicted by the NISQA model and the speech quality score in the speech evaluation dataset. The loss function is used to train the NISQA model until the model converges to obtain a speech quality evaluation model.

[0015] Optionally, the NCSN++ model used for diffusion is used as the Actor model, and a multi-layer Transformer stack is used, followed by an adaptive pooling layer as the Critic model. The NCSN++ model is trained using the speech enhancement dataset and the Actor-Critic algorithm. The speech quality assessment model is used as feedback given by the environment, so that the NCSN++ model is trained with the goal of improving speech quality. For each step in the diffusion, a reward of -1 is given, and the trained NCSN++ model is used as the speech enhancement model to be fine-tuned, specifically including:

[0016] Step S1: Initialize the maximum number of steps L for speech enhancement based on the diffusion model, and randomly initialize the NCSN++ model parameters and the Critic model parameters;

[0017] Step S2: randomly extracting clean speech and noisy speech from the speech enhancement dataset, and using Fourier transform to obtain a clean speech spectrum x and a noisy speech spectrum y;

[0018] Step S3: Initialize time step t to 0;

[0019] Step S4: Record the current noise speech spectrum y and time step t as environment s t , the environment s t Input into the NCSN++ model to obtain the predicted noise spectrum of the current output of the NCSN++ model, which is recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to obtain the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean The final enhanced spectrum result is defined as x. Otherwise, the intermediate spectrum of the current NCSN++ model is defined as x. t+1 / L =xt mean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 ;

[0020] Step S5: Get the current reward r based on the defined external reward and internal reward t , the environment s t and s t+1 and its corresponding actions, provided to the Critic model, using the function Calculate the Critic model loss function L w , use the reverse gradient algorithm to update the current critic model parameters; where γ is the discount factor, is the output of the Critic model at time t, is the output of the critic model at time t+1;

[0021] Step S6: The output of the critic model is inverted as the loss function of the NCSN++ model, and the NCSN++ model parameters are updated using the reverse gradient algorithm;

[0022] Step S7: Repeat steps S4-S6 until t reaches 1, at which point one round of update is completed, and t=1;

[0023] Step S8: Repeat steps S2-S7 and train until the NCSN++ model and the Critic model converge.

[0024] Optionally, the external reward is defined as follows: let L be the maximum number of steps of speech enhancement based on the diffusion model. After the diffusion model enhancement undergoes L steps of denoising, the enhanced speech x0 is obtained. At this time, the NISQA model is used to score x0 to obtain a score. If the score is greater than or equal to 4, the environment gives feedback 2*L. If the score is less than 4, a reward of -L is given. In order to encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each step in the diffusion.

[0025] The internal reward is defined as follows: for a pair of clean speech and noisy speech, let the clean speech signal be x', the noisy speech be y', and the intermediate speech be x mean , x mean Get the formula for x mean =tx′+(1-t)y′, where the time step t is a number that changes from 0 to 1 during the training process. Here, the denoised speech obtained in the tth step of speech enhancement is x1 mean , in the tth step of the training process, the speech quality assessment model calculates x mean mos score scorel and x1 mean If score1 is greater than or equal to score2, a reward of 1 is given; if score1 is less than score2, a reward of -1 is given.

[0026] Optionally, the speech enhancement model to be fine-tuned is used as the Actor model, the Critic model parameters are initialized, the speech enhancement model to be fine-tuned is trained using the unsupervised speech dataset and the Actor-Critic algorithm, and the speech quality assessment model is used as feedback given by the environment, so that the speech enhancement model to be fine-tuned takes improving speech quality as the goal, and a reward of -1 is given for each step in the diffusion. In the process of using the trained speech enhancement model to be fine-tuned as the final speech enhancement model, the current reward r is obtained according to the defined external reward. t ; In the reward definition, even if there is no internal reward, a reward of -1 will still be given for each step.

[0027] In a second aspect, the present application provides a speech enhancement method based on an Actor-Critic algorithm and a diffusion model, the speech enhancement method based on the Actor-Critic algorithm and the diffusion model comprising:

[0028] Obtain a speech evaluation dataset and an unsupervised speech dataset; the speech evaluation dataset includes NISQA, PSTN, and Tencent datasets; the unsupervised speech dataset includes different speech;

[0029] Using the speech evaluation dataset to train the NISQA model, and using the trained NISQA model as a speech quality evaluation model;

[0030] The NCSN++ model used for diffusion is used as the Actor model. A multi-layer Transformer stack followed by an adaptive pooling layer is used as the Critic model. The NCSN++ model is trained using the unsupervised speech dataset and the Actor-Critic algorithm. The speech quality assessment model is used as feedback from the environment to train the NCSN++ model with the goal of improving speech quality. A reward of -1 is given for each step in the diffusion process. The trained NCSN++ model is used as the final speech enhancement model.

[0031] The final speech enhancement model is used to perform speech enhancement on the speech to be enhanced to obtain clean speech.

[0032] Optionally, using the speech evaluation dataset to train a NISQA model, and using the trained NISQA model as a speech quality evaluation model, specifically includes:

[0033] The speech evaluation dataset is used to train the NISQA model. The input of the NISQA model is speech data, the output of the NISQA model is a speech quality evaluation score, and the loss function is the square difference between the score predicted by the NISQA model and the speech quality score in the speech evaluation dataset. The loss function is used to train the NISQA model until the model converges to obtain a speech quality evaluation model.

[0034] Optionally, the NCSN++ model used for diffusion is used as an Actor model, and a multi-layer Transformer stack is used, followed by an adaptive pooling layer as a Critic model. The NCSN++ model is trained using the unsupervised speech dataset and the Actor-Critic algorithm. The speech quality assessment model is used as feedback given by the environment, so that the NCSN++ model is trained with the goal of improving speech quality. For each step in the diffusion, a reward of -1 is given, and the trained NCSN++ model is used as the final speech enhancement model, specifically including:

[0035] Step P1: Initialize the maximum number of steps L for speech enhancement based on the diffusion model, the parameters of the critic model, and randomly initialize the parameters of the NCSN++ model;

[0036] Step P2: randomly extracting speech from the unsupervised speech dataset and obtaining a noise speech spectrum y using Fourier transform;

[0037] Step P3: Initialize time step t to 0;

[0038] Step P4: Record the current noise speech spectrum y and time step t as the environment s t , the environment s t Input into the NCSN++ model to obtain the predicted noise spectrum of the current output of the NCSN++ model, which is recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to obtain the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean The final enhanced spectrum result is defined as x. Otherwise, the intermediate spectrum of the current NCSN++ model is defined as x. t+1 / L =xt mean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 ;

[0039] Step P5: Get the current reward r according to the defined external reward t , the environment s t and s t+1 and its corresponding actions, provided to the Critic model, using the function Calculate the Critic model loss function L w , use the reverse gradient algorithm to update the current critic model parameters; where γ is the discount factor, is the output of the Critic model at time t, is the output of the critic model at time t+1;

[0040] Step P6: Invert the output of the critic model as the loss function of the NCSN++ model, and use the reverse gradient algorithm to update the NCSN++ model parameters;

[0041] Step P7: Repeat steps P4-P6 until t reaches 1, at which point one round of update is completed, and t = 1;

[0042] Step P8: Repeat steps P2-P7 and train until the NCSN++ model and the Critic model converge.

[0043] Optionally, the external reward is defined as follows: let L be the maximum number of steps of speech enhancement based on the diffusion model. After the diffusion model enhancement undergoes L steps of denoising, the enhanced speech x0 is obtained. At this time, the NISQA model is used to score x0 to obtain a score score. If the score is greater than or equal to 4, the environment gives feedback 2*L. If the score is less than 4, a reward of -L is given. In order to encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each step in the diffusion.

[0044] Optionally, in the reward definition, even if there is no intrinsic reward, a reward of -1 is still given at each step.

[0045] According to the specific embodiments provided in this application, this application has the following technical effects:

[0046] This application provides a speech enhancement method based on the Actor-Critic algorithm and a diffusion model. This is the first application of a reinforcement learning algorithm, namely the Actor-Critic algorithm, in a diffusion model-based speech enhancement method. Compared to existing diffusion model-based speech enhancement methods that use specific SDE equations for calculation, this application uses the NCSN++ model used for diffusion as an Actor model to perform the diffusion process. A Critic model with a multi-layer Transformer stack followed by an adaptive pooling layer is used to guide Actor model training. By using the Actor-Critic algorithm to train the NCSN++ model and using a speech quality assessment model as feedback from the environment, the NCSN++ model is trained with the goal of improving speech quality without being restricted by specific SDE equations. This further improves generalization performance and achieves excellent speech enhancement effects in different scenarios. For each diffusion step, a reward of -1 is given to encourage the speech enhancement model to use the fewest possible diffusion steps to achieve efficient speech enhancement, thereby achieving the effect of quickly obtaining clean speech with good quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1 A flowchart of a speech enhancement method based on an Actor-Critic algorithm and a diffusion model provided in Example 1 of the present application;

[0049] Figure 2 A flowchart of a speech enhancement method based on an Actor-Critic algorithm and a diffusion model provided in Example 2 of the present application;

[0050] Figure 3 A schematic diagram of the relationship between the Actor model and the Critic model. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0052] The purpose of this application is to provide a speech enhancement method based on the Actor-Critic algorithm and the diffusion model, which can achieve a good speech enhancement effect in different scenarios and can also obtain clean speech with good quality relatively quickly.

[0053] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0054] Example 1

[0055] like Figure 1 As shown, the speech enhancement method based on the Actor-Critic algorithm and the diffusion model provided in Example 1 of the present application includes:

[0056] Step 101: Acquire a speech enhancement dataset, a speech evaluation dataset, and an unsupervised speech dataset; the speech enhancement dataset includes the VoiceBank-DEMAND dataset; the speech evaluation dataset includes the NISQA, PSTN, and Tencent datasets; and the unsupervised speech dataset includes different speech sounds.

[0057] Step 102: Use the speech evaluation dataset to train the NISQA model, and use the trained NISQA model as the speech quality evaluation model.

[0058] The step 102 specifically includes:

[0059] The NISQA model is trained using a speech evaluation dataset. The input of the NISQA model is speech data, and the output of the NISQA model is the speech quality assessment score. The loss function is the squared difference between the score predicted by the NISQA model and the speech quality score in the speech evaluation dataset. The speech quality assessment model is obtained by training the NISQA model using the loss function until the model converges.

[0060] Step 103: Use the NCSN++ model used for diffusion as the Actor model. Use a multi-layer Transformer stack followed by an adaptive pooling layer as the Critic model. Use the speech enhancement dataset and the Actor-Critic algorithm to train the NCSN++ model. Use the speech quality assessment model as feedback from the environment to train the NCSN++ model with the goal of improving speech quality. Give a reward of -1 for each diffusion step. Use the trained NCSN++ model as the speech enhancement model to be fine-tuned.

[0061] The step 103 specifically includes:

[0062] Step S1: Initialize the maximum number of steps L for speech enhancement based on the diffusion model, and randomly initialize the NCSN++ model parameters and the Critic model parameters.

[0063] Step S2: Randomly extract clean speech and noisy speech from the speech enhancement dataset, and use Fourier transform to obtain the clean speech spectrum x and the noisy speech spectrum y.

[0064] Step S3: Initialize time step t to 0.

[0065] Step S4: Record the current noise speech spectrum y and time step t as environment s t , the environment s t Input into the NCSN++ model to obtain the predicted noise spectrum of the current output of the NCSN++ model, which is recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to obtain the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean The final enhanced spectrum result is defined as x. Otherwise, the intermediate spectrum of the current NCSN++ model is defined as x. t+1 / L =xt mean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 .

[0066] Step S5: Get the current reward r based on the defined external reward and internal rewardt , the environment s t and S t+1 and its corresponding actions, provided to the Critic model, using the function Calculate the Critic model loss function L w , use the reverse gradient algorithm to update the current critic model parameters; where γ is the discount factor, is the output of the Critic model at time t, is the output of the Critic model at time t+1.

[0067] Step S6: The output of the critic model is inverted as the loss function of the NCSN++ model, and the NCSN++ model parameters are updated using the reverse gradient algorithm.

[0068] Step S7: Repeat steps S4-S6 until t reaches 1, at which point one round of update is completed, and t=1.

[0069] Step S8: Repeat steps S2-S7 and train until the NCSN++ model and the Critic model converge.

[0070] The external reward is defined as follows: Let L be the maximum number of steps for speech enhancement based on the diffusion model. After L steps of denoising, the enhanced speech x0 is obtained. The NISQA model is used to score x0 to obtain a score. If the score is greater than or equal to 4, the environment provides feedback of 2*L. If the score is less than 4, a reward of -L is given. To encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each diffusion step.

[0071] The internal reward is defined as follows: For a pair of clean speech and noisy speech, let the clean speech signal be x', the noisy speech be y', and the intermediate speech be x mean , x mean Get the formula for x mean =tx′+(1-t)y′, where the time step t is a number that changes from 0 to 1 during the training process. Here, the denoised speech obtained in the tth step of speech enhancement is x1 mean , in the tth step of the training process, the speech quality assessment model calculates x mean mos score scorel and x1 mean If score1 is greater than or equal to score2, a reward of 1 is given; if score1 is less than score2, a reward of -1 is given.

[0072] Step 104: Use the speech enhancement model to be fine-tuned as the Actor model, initialize the Critic model parameters, use the unsupervised speech dataset and the Actor-Critic algorithm to train the speech enhancement model to be fine-tuned, use the speech quality assessment model as feedback given by the environment, and make the speech enhancement model to be fine-tuned take improving speech quality as the goal. For each step in the diffusion, give a reward of -1, and use the trained speech enhancement model to be fine-tuned as the final speech enhancement model.

[0073] In the process of step 104, the current reward r is obtained according to the defined external reward. t ; In the reward definition, even if there is no internal reward, a reward of -1 will still be given for each step.

[0074] Step 105: Use the final speech enhancement model to perform speech enhancement on the speech to be enhanced to obtain clean speech.

[0075] The following is a specific example to illustrate the technical solution of the speech enhancement method based on the Actor-Critic algorithm and the diffusion model provided in Example 1 of the present application:

[0076] Step 1: Dataset collection, training data set: This part is divided into:

[0077] 1. Speech enhancement dataset: Collect data with clean speech and noisy speech pairs. This can be collected through datasets publicly available on the Internet, such as the voicebank-demand dataset (i.e., the VoiceBank-DEMAND dataset).

[0078] 2. Speech evaluation dataset: Collect non-embedded speech evaluation datasets, such as NISQA, PSTN, and Tencent datasets. These datasets all have speech and corresponding human subjective scores, namely MOS scores.

[0079] 3. Unsupervised speech dataset: Collect speech data from the Internet, etc., and do not require a clean dataset.

[0080] Step 2: Train the speech quality assessment model.

[0081] The NISQA model is selected as the model and trained using the speech evaluation dataset collected in step 1. The model input is speech data and the output is the speech quality evaluation score. The loss function is the square difference between the predicted score and the speech quality score in the dataset, that is:

[0082] loss=||pred-target|| 2 ;

[0083] Among them, pred is the model output and target is the real speech quality assessment score.

[0084] Use the loss function to train until the model converges to obtain the speech quality assessment model.

[0085] Step 3: Use the Actor-Critic algorithm to train the speech enhancement model. Before starting training, define the following elements first:

[0086] Rewards: Rewards are divided into two parts: external rewards and internal rewards:

[0087] External reward: Let L be the maximum number of steps for speech enhancement based on the diffusion model. After L steps of denoising, the enhanced speech x0 is obtained. The NISQA model is used to score x0 to obtain a score. If the score is greater than or equal to 4, the environment provides feedback of 2*L. If the score is less than 4, a reward of -L is given. To encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each diffusion step.

[0088] Internal reward: For a pair of clean speech and noisy speech, let the clean speech signal be x' and the noisy speech be y'. Get the intermediate speech as x mean , x mean The formula for obtaining it is as follows:

[0089] x mean =tx'+(1-t)y';

[0090] Among them, t is the time step, which is a number that changes from 0 to 1 during the training process. Here, the denoised speech obtained in the tth step of speech enhancement is recorded as x1 mean , in the tth step of the training process, the NISQA calculation x obtained in step 2 is used mean and x1 mean The mos scores score1 and score2, if score1 is greater than or equal to score2, then reward 1 is given, if score1<score2: reward -1 is given.

[0091] Model:

[0092] Actor model: The NCSN++ model is used as the speech enhancement model. The model inputs the time step t and the noisy speech spectrum (obtained by Fourier transforming the noisy speech) and outputs the predicted noise spectrum.

[0093] Critic model: A multi-layer transformer stack followed by an adaptive pooling layer is used as the model, hereinafter referred to as V w, where w represents the parameters of the current critic model. The input is the noisy speech at each time step (in the specific training step, the noisy speech at each time step is obtained by performing an inverse Fourier transform on the intermediate spectrum) and the output of the actor model. The output is the expected reward for the actor model to perform the action at the current time step (expected reward is a score for the action, a concept in reinforcement learning that provides an overall evaluation of the entire denoising process).

[0094] The above discussion of rewards and model definitions is all part of the definition process, primarily to clarify some details in the training process below. The NISQA model is used as part of the reward allocation in this application. NCSN++ is defined as an actor model and critic in the above text. These are all required for reinforcement learning training using the actor-critic reinforcement learning algorithm. The final speech enhancement model consists solely of the NCSN++ model, which is also an actor model.

[0095] The specific training process for a speech enhancement model with clean speech supervision (the only speech enhancement model with clean speech supervision is the actor model. The actor model and the NCSN++ model are the same, except that in the reinforcement learning framework, that is, the actor-critic framework, the model is called an actor model, but the specific implementation is the NCSN++ model) is as follows:

[0096] (1) Initialize the maximum number of steps L for speech enhancement based on the diffusion model, and randomly initialize the speech enhancement model parameters and the critic model parameters.

[0097] (2) Select the speech enhancement dataset in step 1, randomly extract clean speech and noisy speech from the dataset (i.e., randomly extract any pair of clean speech and noisy speech), and use Fourier transform to obtain the clean speech spectrum x and the noisy speech spectrum y.

[0098] (3) Initialize time step t to 0.

[0099] (4) The current noise speech spectrum y and time step t are recorded as environment s t , the environment s t Input into the Actor model defined in the above steps to obtain the predicted noise spectrum of the Actor's current output, recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to obtain the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean The final enhanced spectrum result is defined as x. Otherwise, the intermediate result spectrum of the current speech enhancement model is defined as x. t+1 / L =xtmean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 (by passing the environment s t+1 Input into the NCSN++ model and get the action a output by the NCSN++ model t+1 ). Where t is the current time step, 1 / L represents the step size, and t+1 / L is the updated time step.

[0100] (5) According to the external rewards and internal rewards defined above, obtain the current reward r t , the environment s t and s t+1 And its corresponding action, provided to critic, use the following function to calculate the critic model loss function L w :

[0101]

[0102] Use the reverse gradient algorithm to update the current critic model parameters. Among them, γ is the discount factor used to weigh the immediate reward and long-term reward, which is set to 0.9. is the output of the Critic model at time t, that is, the score at time t, It is the output of the Critic model at time t+1, that is, the score at time t+1.

[0103] (6) Calculate the loss function of the actor model: the output of the critic model Take the inverse as the loss function L of the actor model actor ,Right now:

[0104]

[0105] Update the actor model parameters using the back-gradient algorithm.

[0106] (7) Repeat steps (4)-(6) until t reaches 1, at which point one round of update is completed, and t=1.

[0107] (8) Repeat steps (2)-(7) until the model (i.e., actor model and critic model) converges. Convergence condition: If the loss function value does not decrease significantly after multiple consecutive iterations, the model can be considered to have converged.

[0108] Fine-tuning steps:

[0109] For the speech enhancement model without clean speech supervision (the model without clean speech supervision and the model with clean speech supervision have the same model composition, but lack internal rewards during training), training is as follows:

[0110] (1) Initialize the maximum number of steps L for speech enhancement based on the diffusion model and the critic model parameters. If fine-tuning the model, initialize the speech enhancement model (NCSN++\actor model) parameters to the model parameters trained with the supervised dataset above.

[0111] (2) Randomly extract speech from the unsupervised speech dataset and use Fourier transform to obtain the noisy speech spectrum y.

[0112] (3) Initialize time step t to 0.

[0113] (4) The current noise speech spectrum y and time step t are recorded as environment s t , the environment s t Input into the Actor model defined in the above steps to obtain the predicted noise spectrum of the Actor's current output, recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to obtain the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean The final enhanced spectrum result is defined as x. Otherwise, the intermediate result spectrum of the current speech enhancement model is defined as x. t+1 / L =xt mean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 .

[0114] (5) According to the external reward defined above (in the reward definition, even if there is no internal reward, each step will still be given a reward of -1, which is to encourage the use of fewer steps), get the current reward r t , the environment s t and s t+1 And its corresponding action, provided to critic, use the following function to calculate the critic model loss function L w :

[0115]

[0116] Use the reverse gradient algorithm to update the current critic model parameters. Where γ is the discount factor, is the output of the Critic model at time t, is the output of the Critic model at time t+1.

[0117] (6) Calculate the loss function of the actor model: the output of the critic model Take the inverse as the loss function L of the actor model actor ,Right now:

[0118]

[0119] Update the actor model parameters using the back-gradient algorithm.

[0120] (7) Repeat steps (4)-(6) until t reaches 1, at which point one round of update is completed, and t=1.

[0121] (8) Repeat steps (2)-(7) until the model (i.e., the critic and actor models) converges. Convergence condition: If the loss function value does not decrease significantly after multiple consecutive iterations, the model can be considered to have converged.

[0122] Subsequent use:

[0123] In the clean speech supervision, a preliminary speech enhancement model will be trained. Subsequently, unsupervised data will be used for speech enhancement training to enrich the training data set. At the same time, since clean speech supervision is not required, this method is relatively cheap. For various scenarios, after defining the number of diffusion steps, the noisy audio and time step t (initialized to 0) can be directly provided to the speech enhancement model for iterative denoising. The specific steps are as (3)-(4) in the training process in step 3 above to achieve the purpose of speech enhancement. When the model performs poorly in an unknown field, the above-mentioned unsupervised speech enhancement training method (i.e., the speech enhancement model training process without clean speech supervision) can be used to fine-tune the model (i.e., fine-tune the actor model parameters) without the need for clean speech.

[0124] The training of the speech enhancement method based on the Actor-Critic algorithm and the diffusion model provided in Example 1 of the present application starts with using clean and noisy speech pairs, but what is more important is actually the part of the unsupervised data set. This method expands the data set for training the speech enhancement model without increasing the cost too much.

[0125] Example 2

[0126] like Figure 2 As shown, the speech enhancement method based on the Actor-Critic algorithm and the diffusion model provided in Example 2 of the present application includes:

[0127] Step 201: Acquire a speech evaluation dataset and an unsupervised speech dataset; the speech evaluation dataset includes NISQA, PSTN, and Tencent datasets; the unsupervised speech dataset includes different speech.

[0128] Step 202: Use the speech evaluation dataset to train the NISQA model, and use the trained NISQA model as the speech quality evaluation model.

[0129] The step 202 specifically includes:

[0130] The NISQA model is trained using a speech evaluation dataset. The input of the NISQA model is speech data, and the output of the NISQA model is the speech quality assessment score. The loss function is the squared difference between the score predicted by the NISQA model and the speech quality score in the speech evaluation dataset. The speech quality assessment model is obtained by training the NISQA model using the loss function until the model converges.

[0131] Step 203: The NCSN++ model used for diffusion is used as the Actor model. A multi-layer Transformer stack is used, followed by an adaptive pooling layer as the Critic model. The NCSN++ model is trained using an unsupervised speech dataset and the Actor-Critic algorithm. A speech quality assessment model is used as feedback from the environment to train the NCSN++ model with the goal of improving speech quality. A reward of -1 is given for each step in the diffusion process. The trained NCSN++ model is used as the final speech enhancement model.

[0132] The step 203 specifically includes:

[0133] Step P1: Initialize the maximum number of steps L and the critic model parameters of the diffusion model-based speech enhancement, and randomly initialize the NCSN++ model parameters.

[0134] Step P2: Randomly extract speech from the unsupervised speech dataset and use Fourier transform to obtain the noisy speech spectrum y.

[0135] Step P3: Initialize time step t to 0.

[0136] Step P4: Record the current noise speech spectrum y and time step t as the environment s t , the environment s t Input into the NCSN++ model to obtain the predicted noise spectrum of the current output of the NCSN++ model, which is recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to obtain the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt meanThe final enhanced spectrum result is defined as x. Otherwise, the intermediate spectrum of the current NCSN++ model is defined as x. t+1 / L =xt mean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 .

[0137] Step P5: Get the current reward r according to the defined external reward t , the environment s t and s t+1 and its corresponding actions, provided to the Critic model, using the function Calculate the Critic model loss function L w , use the reverse gradient algorithm to update the current critic model parameters; where γ is the discount factor, is the output of the Critic model at time t, is the output of the Critic model at time t+1.

[0138] The external reward is defined as follows: Let L be the maximum number of steps for speech enhancement based on the diffusion model. After L steps of denoising, the enhanced speech x0 is obtained. The NISQA model is used to score x0 to obtain a score. If the score is greater than or equal to 4, the environment provides feedback of 2*L. If the score is less than 4, a reward of -L is given. To encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each diffusion step.

[0139] In the reward definition, even if there is no intrinsic reward, a reward of -1 is still given for each step.

[0140] Step P6: Invert the output of the Critic model as the loss function of the NCSN++ model, and use the reverse gradient algorithm to update the NCSN++ model parameters.

[0141] Step P7: Repeat steps P4-P6 until t reaches 1, at which point one round of update is completed, and t=1.

[0142] Step P8: Repeat steps P2-P7 and train until the NCSN++ model and the Critic model converge.

[0143] Step 204: Use the final speech enhancement model to perform speech enhancement on the speech to be enhanced to obtain clean speech.

[0144] The following is a specific example to illustrate the technical solution of the speech enhancement method based on the Actor-Critic algorithm and the diffusion model provided in Example 2 of this application:

[0145] Step 1: Dataset collection, training data set: This part is divided into:

[0146] 1. Speech evaluation dataset: Collect non-embedded speech evaluation datasets, such as NISQA, PSTN, and Tencent datasets. These datasets all have speech and corresponding human subjective scores, namely MOS scores.

[0147] 2. Unsupervised speech dataset: Collect speech data from the Internet, etc., and do not require a clean dataset.

[0148] Step 2: Train the speech quality assessment model.

[0149] The NISQA model is selected as the model and trained using the speech evaluation dataset collected in step 1. The model input is speech data and the output is the speech quality evaluation score. The loss function is the square difference between the predicted score and the speech quality score in the dataset, that is:

[0150] loss=||pred-target|| 2 ;

[0151] Among them, pred is the model output and target is the real speech quality assessment score.

[0152] Use the loss function to train until the model converges to obtain the speech quality assessment model.

[0153] Step 3: Use the Actor-Critic algorithm to train the speech enhancement model. Before starting training, define the following elements first:

[0154] award:

[0155] External reward: Let L be the maximum number of steps for speech enhancement based on the diffusion model. After L steps of denoising, the enhanced speech x0 is obtained. The NISQA model is used to score x0 to obtain a score. If the score is greater than or equal to 4, the environment provides feedback of 2*L. If the score is less than 4, a reward of -L is given. To encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each diffusion step.

[0156] Model:

[0157] Actor model: The NCSN++ model is used as the speech enhancement model. The model inputs the time step t and the noisy speech spectrum (obtained by Fourier transforming the noisy speech) and outputs the predicted noise spectrum.

[0158] Critic model: A multi-layer transformer stack followed by an adaptive pooling layer is used as the model, hereinafter referred to as V w , where w represents the parameters of the current critic model. The input is the noisy speech at each time step (in the specific training step, the noisy speech at each time step is obtained by performing an inverse Fourier transform on the intermediate spectrum) and the output of the actor model. The output is the expected reward for the actor model to perform the action at the current time step (expected reward is a score for the action, a concept in reinforcement learning that provides an overall evaluation of the entire denoising process).

[0159] The above discussion of rewards and model definitions is all part of the definition process, primarily to clarify some details in the training process below. The NISQA model is used as part of the reward allocation in this application. NCSN++ is defined as an actor model and critic in the above text. These are all required for reinforcement learning training using the actor-critic reinforcement learning algorithm. The final speech enhancement model consists solely of the NCSN++ model, which is also an actor model.

[0160] For a speech enhancement model without clean speech supervision (different from the speech enhancement method based on the Actor-Critic algorithm and the diffusion model provided in Example 1 and the speech enhancement method based on the Actor-Critic algorithm and the diffusion model provided in Example 2, it is also possible not to use a supervised dataset, i.e., a speech enhancement dataset, but to directly use an unsupervised dataset to train the speech enhancement model without clean speech supervision) training is as follows:

[0161] (1) Initialize the maximum number of steps L for speech enhancement based on the diffusion model and the critic model parameters. If the speech enhancement model is directly trained without clean speech supervision, the parameters of the speech enhancement model are randomly initialized.

[0162] (2) Randomly extract speech from the unsupervised speech dataset and use Fourier transform to obtain the noisy speech spectrum y.

[0163] (3) Initialize time step t to 0.

[0164] (4) The current noise speech spectrum y and time step t are recorded as environment s t , the environment s tInput into the Actor model defined in the above steps to obtain the predicted noise spectrum of the Actor's current output, recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to obtain the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean The final enhanced spectrum result is defined as x. Otherwise, the intermediate result spectrum of the current speech enhancement model is defined as x. t+1 / L =xt mean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 (by passing the environment s t+1 Input into the NCSN++ model and get the action a output by the NCSN++ model t+1 ). Where t is the current time step, 1 / L represents the step size, and t+1 / L is the updated time step.

[0165] (5) According to the external reward defined above (in the reward definition, even if there is no internal reward, each step will still be given a reward of -1, which is to encourage the use of fewer steps), get the current reward r t , the environment s t and s t+1 And its corresponding action, provided to critic, use the following function to calculate the critic model loss function L w :

[0166]

[0167] Use the reverse gradient algorithm to update the current critic model parameters. Among them, γ is the discount factor used to weigh the immediate reward and long-term reward, which is set to 0.9. is the output of the Critic model at time t, that is, the score at time t, It is the output of the Critic model at time t+1, that is, the score at time t+1.

[0168] (6) Calculate the loss function of the actor model: the output of the critic model Take the inverse as the loss function L of the actor model actor ,Right now:

[0169]

[0170] Update the actor model parameters using the back-gradient algorithm.

[0171] (7) Repeat steps (4)-(6) until t reaches 1, at which point one round of update is completed, and t=1.

[0172] (8) Repeat steps (2)-(7) until the model (i.e., the critic and actor models) converges. Convergence condition: If the loss function value does not decrease significantly after multiple consecutive iterations, the model can be considered to have converged.

[0173] This application is a combination of diffusion model and reinforcement learning. Reinforcement learning uses the actor-critic framework. The relationship between the actor model and the critic model is as follows: Figure 3 As shown, the actor model is an enhanced model, specifically implemented as the NCSN++ model. The diffusion model in this application is not a specific model, but a method with certain generalization performance, and this application further improves its generalization performance.

[0174] Currently, most research on speech enhancement focuses on processing the speech signal of a single speaker, typically using pure speech recorded in a laboratory environment, and tends to be more theoretical. However, in practical applications, speech signals are often affected by various noise and interference, such as background noise, echo, and reverberation. This places demands on the generalization performance of existing speech enhancement models. Diffusion models have addressed this issue to a certain extent, but the following problems still exist: 1. Existing diffusion models rely on predefined SDE equations, etc. These assumptions limit the generalization performance of model scores to a certain extent. 2. Speech enhancement based on diffusion models requires multiple rounds of iterative denoising during inference, which is time-consuming. To address the above issues, this application uses reinforcement learning (Actor-sritic algorithm is a type of reinforcement learning algorithm) for model training, which does not rely on a specific SDE equation. At the same time, because the research is based on the diffusion model, the model trained in this application retains the generalization performance of the diffusion model. In addition, because the reward function is used during training to encourage the speech enhancement model to use fewer diffusion steps, the trained speech enhancement model can also reduce the number of iterations, achieving rapid speech enhancement.

[0175] Compared with existing speech enhancement methods, this application is also a speech enhancement method based on a diffusion model, but it is improved on the basis of the existing speech enhancement method based on the diffusion model, that is, the reinforcement learning algorithm actor-critic algorithm is combined with the diffusion model, which avoids the need for speech pairs to a certain extent and does not require the assumption of the SDE equation.

[0176] Compared with existing speech enhancement methods, this application does not rely on various assumptions such as predefined SDE formulas, and is highly generalizable and improves inference speed, mainly due to:

[0177] 1. The actor-critic algorithm is used to train the model, and the speech quality assessment model is used as the feedback given by the environment. This allows the model training to improve speech quality without being restricted by the SDE equation.

[0178] 2. Model training is based on a diffusion model (i.e., a diffusion model with multiple noise addition and denoising methods), retaining the generalization performance of the diffusion model itself (the diffusion model itself has generalization performance, but when relying on clean noise speech pairs, the generalization performance is still somewhat limited. This application uses a reinforcement learning algorithm, the actor-critc algorithm, to further improve generalization). At the same time, the model can be trained without clean speech supervision, which greatly reduces the cost and price of migrating the model to unknown areas.

[0179] 3. In the reward, a -1 reward is given for each diffusion step to encourage the speech enhancement model to use as few diffusion steps as possible to achieve efficient speech enhancement.

[0180] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A speech enhancement method based on Actor-Critic algorithm and diffusion model, characterized in that: The speech enhancement method based on the Actor-Critic algorithm and the diffusion model includes: Acquire a speech enhancement dataset, a speech evaluation dataset, and an unsupervised speech dataset; the speech enhancement dataset includes the VoiceBank-DEMAND dataset; the speech evaluation dataset includes the NISQA, PSTN, and Tencent datasets; and the unsupervised speech dataset includes different speech sounds; Using the speech evaluation dataset to train the NISQA model, and using the trained NISQA model as a speech quality evaluation model; The NCSN++ model used for diffusion is used as the Actor model. A multi-layer Transformer stack followed by an adaptive pooling layer is used as the Critic model. The NCSN++ model is trained using the speech enhancement dataset and the Actor-Critic algorithm. The speech quality assessment model is used as feedback from the environment to train the NCSN++ model with the goal of improving speech quality. A reward of -1 is given for each step in the diffusion process. The trained NCSN++ model is used as the speech enhancement model to be fine-tuned. The speech enhancement model to be fine-tuned is used as the Actor model, the Critic model parameters are initialized, the speech enhancement model to be fine-tuned is trained using the unsupervised speech dataset and the Actor-Critic algorithm, the speech quality assessment model is used as feedback given by the environment, and the speech enhancement model to be fine-tuned is guided to improve speech quality as the goal, a reward of -1 is given for each step in the diffusion process, and the trained speech enhancement model to be fine-tuned is used as the final speech enhancement model; The final speech enhancement model is used to perform speech enhancement on the speech to be enhanced to obtain clean speech.

2. The speech enhancement method based on the Actor-Critic algorithm and the diffusion model according to claim 1, characterized in that: The NISQA model is trained using the speech evaluation dataset, and the trained NISQA model is used as a speech quality evaluation model, specifically including: The speech evaluation dataset is used to train the NISQA model. The input of the NISQA model is speech data, the output of the NISQA model is a speech quality evaluation score, and the loss function is the square difference between the score predicted by the NISQA model and the speech quality score in the speech evaluation dataset. The loss function is used to train the NISQA model until the model converges to obtain a speech quality evaluation model.

3. The speech enhancement method based on the Actor-Critic algorithm and the diffusion model according to claim 1, characterized in that: The NCSN++ model used for diffusion is used as the Actor model. A multi-layer Transformer stack followed by an adaptive pooling layer is used as the Critic model. The NCSN++ model is trained using the speech enhancement dataset and the Actor-Critic algorithm. The speech quality assessment model is used as feedback from the environment to train the NCSN++ model with the goal of improving speech quality. A reward of -1 is given for each step in the diffusion process. The trained NCSN++ model is used as the speech enhancement model to be fine-tuned. Specifically, the following steps are involved: Step S1: Initialize the maximum number of steps L for speech enhancement based on the diffusion model, and randomly initialize the NCSN++ model parameters and the Critic model parameters; Step S2: randomly extracting clean speech and noisy speech from the speech enhancement dataset, and using Fourier transform to obtain a clean speech spectrum x and a noisy speech spectrum y; Step S3: Initialize time step t to 0; Step S4: Record the current noise speech spectrum y and time step t as environment s t , the environment s t Input into the NCSN++ model to obtain the predicted noise spectrum of the current output of the NCSN++ model, which is recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to obtain the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean The final enhanced spectrum result is defined as x. Otherwise, the intermediate spectrum of the current NCSN++ model is defined as x. t+1 / L =xt mean +noiuse, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 ; Step S5: Get the current reward r based on the defined external reward and internal reward t , the environment s t and s t+1 and its corresponding actions, provided to the Critic model, using the function Calculate the Critic model loss function L w , use the reverse gradient algorithm to update the current critic model parameters; where γ is the discount factor, is the output of the Critic model at time t, is the output of the critic model at time t+1; Step S6: The output of the critic model is inverted as the loss function of the NCSN++ model, and the NCSN++ model parameters are updated using the reverse gradient algorithm; Step S7: Repeat steps S4-S6 until t reaches 1, at which point one round of update is completed, and t=1; Step S8: Repeat steps S2-S7 and train until the NCSN++ model and the Critic model converge.

4. The method for speech enhancement based on the Actor-Critic algorithm and the diffusion model according to claim 3, characterized in that: The external reward is defined as follows: Let L be the maximum number of steps for speech enhancement based on the diffusion model. After L steps of denoising, the enhanced speech x0 is obtained. The NISQA model is used to score x0 to obtain a score. If the score is greater than or equal to 4, the environment gives feedback of 2*L. If the score is less than 4, a reward of -L is given. To encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each step in the diffusion process. The internal reward is defined as follows: for a pair of clean speech and noisy speech, let the clean speech signal be x', the noisy speech be y', and the intermediate speech be x mean , x mean Get the formula for x mean =tx'+(1-t)y', where time step t is a number that changes from 0 to 1 during the training process. Here, the denoised speech obtained in the tth step of speech enhancement is x1 mean , in the tth step of the training process, the speech quality assessment model calculates x mean mos score score1 and x1 mean If score1 is greater than or equal to score2, a reward of 1 is given; if score1 is less than score2, a reward of -1 is given.

5. The method for speech enhancement based on the Actor-Critic algorithm and the diffusion model according to claim 4, characterized in that: The speech enhancement model to be fine-tuned is used as the Actor model, the Critic model parameters are initialized, the speech enhancement model to be fine-tuned is trained using the unsupervised speech dataset and the Actor-Critic algorithm, and the speech quality assessment model is used as the feedback given by the environment to make the speech enhancement model to be fine-tuned take improving speech quality as the goal. For each step in the diffusion, a reward of -1 is given. In the process of using the trained speech enhancement model to be fine-tuned as the final speech enhancement model, the current reward r is obtained according to the defined external reward. t ; In the reward definition, even if there is no internal reward, a reward of -1 will still be given for each step.

6. A speech enhancement method based on Actor-Critic algorithm and diffusion model, characterized in that: The speech enhancement method based on the Actor-Critic algorithm and the diffusion model includes: Obtain a speech evaluation dataset and an unsupervised speech dataset; the speech evaluation dataset includes NISQA, PSTN, and Tencent datasets; the unsupervised speech dataset includes different speech; Using the speech evaluation dataset to train the NISQA model, and using the trained NISQA model as a speech quality evaluation model; The NCSN++ model used for diffusion is used as the Actor model. A multi-layer Transformer stack followed by an adaptive pooling layer is used as the Critic model. The NCSN++ model is trained using the unsupervised speech dataset and the Actor-Critic algorithm. The speech quality assessment model is used as feedback from the environment to train the NCSN++ model with the goal of improving speech quality. A reward of -1 is given for each step in the diffusion process. The trained NCSN++ model is used as the final speech enhancement model. The final speech enhancement model is used to perform speech enhancement on the speech to be enhanced to obtain clean speech.

7. The method for speech enhancement based on the Actor-Critic algorithm and the diffusion model according to claim 6, characterized in that: The NISQA model is trained using the speech evaluation dataset, and the trained NISQA model is used as a speech quality evaluation model, specifically including: The speech evaluation dataset is used to train the NISQA model. The input of the NISQA model is speech data, the output of the NISQA model is a speech quality evaluation score, and the loss function is the square difference between the score predicted by the NISQA model and the speech quality score in the speech evaluation dataset. The loss function is used to train the NISQA model until the model converges to obtain a speech quality evaluation model.

8. The method for speech enhancement based on the Actor-Critic algorithm and the diffusion model according to claim 6, characterized in that: The NCSN++ model used for diffusion is used as the Actor model. A multi-layer Transformer stack is used, followed by an adaptive pooling layer as the Critic model. The NCSN++ model is trained using the unsupervised speech dataset and the Actor-Critic algorithm. The speech quality assessment model is used as feedback from the environment to train the NCSN++ model with the goal of improving speech quality. A reward of -1 is given for each step in the diffusion process. The trained NCSN++ model is used as the final speech enhancement model. Specifically, the following steps are involved: Step P1: Initialize the maximum number of steps L for speech enhancement based on the diffusion model, the parameters of the critic model, and randomly initialize the parameters of the NCSN++ model; Step P2: randomly extracting speech from the unsupervised speech dataset and obtaining a noise speech spectrum y using Fourier transform; Step P3: Initialize time step t to 0; Step P4: Record the current noise speech spectrum y and time step t as the environment s t , the environment s t Input into the NCSN++ model to obtain the predicted noise spectrum of the current output of the NCSN++ model, which is recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to obtain the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean The final enhanced spectrum result is defined as x. Otherwise, the intermediate spectrum of the current NCSN++ model is defined as x. t+1 / L =xt mean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 ; Step P5: Get the current reward r according to the defined external reward t , the environment s t and s t+1 and its corresponding actions, provided to the Critic model, using the function Calculate the Critic model loss function L w , use the reverse gradient algorithm to update the current critic model parameters; where γ is the discount factor, is the output of the Critic model at time t, is the output of the critic model at time t+1; Step P6: Invert the output of the critic model as the loss function of the NCSN++ model, and use the reverse gradient algorithm to update the NCSN++ model parameters; Step P7: Repeat steps P4-P6 until t reaches 1, at which point one round of update is completed, and t = 1; Step P8: Repeat steps P2-P7 and train until the NCSN++ model and the Critic model converge.

9. The method for speech enhancement based on the Actor-Critic algorithm and the diffusion model according to claim 8, characterized in that: The external reward is defined as follows: Let L be the maximum number of steps for speech enhancement based on the diffusion model. After L steps of denoising, the enhanced speech x0 is obtained. The NISQA model is used to score x0 to obtain a score. If the score is greater than or equal to 4, the environment gives feedback of 2*L. If the score is less than 4, a reward of -L is given. To encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each diffusion step.

10. The method for speech enhancement based on the Actor-Critic algorithm and the diffusion model according to claim 9, characterized in that: In the reward definition, even if there is no intrinsic reward, a reward of -1 is still given for each step.

Images