Speech enhancement method based on Actor-Critic algorithm and diffusion model

Abstract

The invention discloses a speech enhancement method based on an Actor-Critic algorithm and a diffusion model, and relates to the field of speech enhancement, and the method comprises the steps: taking an NCSN + + model used for diffusion as an Actor model, training the model through employing a speech enhancement data set and the Actor-Critic algorithm, employing a speech quality evaluation model as the feedback given by an environment, and carrying out the recognition of the Actor-Critic algorithm. The method comprises the following steps: carrying out diffusion on a speech enhancement model, enabling model training to take improvement of speech quality as a target, giving an award of-1 for each step in diffusion, taking the trained model as a speech enhancement model to be finely tuned, taking the model as an Actor model, training the model by using an unsupervised speech data set and an Actor-Critic algorithm, realizing fine tuning of the model, and further obtaining a final speech enhancement model. According to the invention, a good speech enhancement effect can be obtained in different scenes, and clean speech with good quality can be obtained rapidly.

Description

Technical Field

[0001] The present application relates to the technical field of speech enhancement, and in particular to a speech enhancement method based on an Actor-Critic algorithm and a diffusion model. Background Art

[0002] At present, most of the research on speech enhancement focuses on processing the speech signal of a single speaker, usually pure speech recorded in a laboratory environment, and is more inclined to theoretical research. However, in practical applications, speech signals are often affected by various noises and interferences, such as background noise, echo, reverberation, etc., which puts forward requirements on the generalization performance of existing speech enhancement models.

[0003] Speech enhancement technology faces complex application scenarios, and traditional speech enhancement methods show obvious limitations in these scenarios. Whenever the application field changes, the speech enhancement model needs to be fine-tuned to adapt to the new environment. In recent years, speech enhancement methods based on diffusion models have solved this problem to a certain extent, but their training still relies on a large number of clean and noisy corpus pairs and specific SDE equations, which limits the generalization performance of the diffusion model to a certain extent. At the same time, the speech enhancement model based on the diffusion model has the problem of long inference time.

[0004] In summary, speech enhancement based on diffusion models has the problems of limited generalization performance and long inference time. Therefore, the existing speech enhancement methods based on diffusion models cannot achieve a good speech enhancement effect in different scenarios, and cannot obtain clean speech with good quality relatively quickly. Summary of the invention

[0005] The purpose of this application is to provide a speech enhancement method based on the Actor-Critic algorithm and the diffusion model, which can achieve a good speech enhancement effect in different scenarios and can also obtain clean speech with good quality relatively quickly.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a speech enhancement method based on an Actor-Critic algorithm and a diffusion model, and the speech enhancement method based on the Actor-Critic algorithm and the diffusion model comprises:

[0008] Acquire a speech enhancement dataset, a speech evaluation dataset, and an unsupervised speech dataset; the speech enhancement dataset includes a VoiceBank-DEMAND dataset; the speech evaluation dataset includes a NISQA, PSTN, and Tencent dataset; and the unsupervised speech dataset includes different voices;

[0009] Using the speech evaluation data set to train a NISQA model, and using the trained NISQA model as a speech quality evaluation model;

[0010] The NCSN++ model used for diffusion is used as the Actor model, and a multi-layer Transformer stack is used, followed by an adaptive pooling layer as the Critic model. The NCSN++ model is trained using the speech enhancement dataset and the Actor-Critic algorithm. The speech quality assessment model is used as feedback given by the environment to train the NCSN++ model with the goal of improving speech quality. For each step in the diffusion, a reward of -1 is given, and the trained NCSN++ model is used as the speech enhancement model to be fine-tuned.

[0011] The speech enhancement model to be fine-tuned is used as the Actor model, the Critic model parameters are initialized, the speech enhancement model to be fine-tuned is trained using the unsupervised speech data set and the Actor-Critic algorithm, the speech quality assessment model is used as feedback given by the environment, so that the speech enhancement model to be fine-tuned takes improving speech quality as the goal, a reward of -1 is given for each step in the diffusion, and the trained speech enhancement model to be fine-tuned is used as the final speech enhancement model;

[0012] The final speech enhancement model is used to perform speech enhancement on the speech to be enhanced to obtain clean speech.

[0013] Optionally, using the speech evaluation data set to train a NISQA model, and using the trained NISQA model as a speech quality evaluation model, specifically includes:

[0014] The speech evaluation dataset is used to train the NISQA model, the input of the NISQA model is speech data, the output of the NISQA model is a speech quality evaluation score, the loss function is the square difference between the score predicted by the NISQA model and the speech quality score in the speech evaluation dataset, and the loss function is used to train the NISQA model until it converges to obtain a speech quality evaluation model.

[0015] Optionally, the NCSN++ model used for diffusion is used as an Actor model, and a multi-layer Transformer stack is used, followed by an adaptive pooling layer as a Critic model. The NCSN++ model is trained using the speech enhancement dataset and the Actor-Critic algorithm, and the speech quality assessment model is used as feedback given by the environment to train the NCSN++ model to improve speech quality. For each step in the diffusion, a reward of -1 is given, and the trained NCSN++ model is used as the speech enhancement model to be fine-tuned, specifically including:

[0016] Step S1: Initialize the maximum number of steps L of speech enhancement based on the diffusion model, and randomly initialize the NCSN++ model parameters and the Critic model parameters;

[0017] Step S2: randomly extracting clean speech and noisy speech from the speech enhancement data set, and using Fourier transform to obtain a clean speech spectrum x and a noisy speech spectrum y;

[0018] Step S3: Initialize time step t to 0;

[0019] Step S4: Record the current noise speech spectrum y and time step t as environment s t , the environment s t Input into the NCSN++ model to obtain the predicted noise spectrum of the current output of the NCSN++ model, recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to get the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean is the final enhanced spectrum result, otherwise the intermediate result spectrum of the current NCSN++ model is defined as x t+1 / L =xt mean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 ;

[0020] Step S5: Get the current reward r according to the defined external reward and internal reward t , the environment s t and t+1 and its corresponding actions, provided to the Critic model, using the function Calculate the Critic model loss function L w , use the reverse gradient algorithm to update the current Critic model parameters; where γ is the discount factor, is the output of the Critic model at time t, is the output of the Critic model at time t+1;

[0021] Step S6: The output of the Critic model is inverted as the loss function of the NCSN++ model, and the NCSN++ model parameters are updated using the reverse gradient algorithm;

[0022] Step S7: Repeat steps S4-S6 until t reaches 1, at which point one round of update is completed, and t=1;

[0023] Step S8: Repeat steps S2-S7 and train until the NCSN++ model and the Critic model converge.

[0024] Optionally, the external reward is defined as follows: let L be the maximum number of steps of speech enhancement based on the diffusion model. After the diffusion model enhancement undergoes L steps of denoising, the enhanced speech x0 is obtained. At this time, the NISQA model is used to score x0 to obtain a score score. If the score is greater than or equal to 4, the environment gives feedback 2*L. If the score is less than 4, a reward of -L is given. In order to encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each step in the diffusion.

[0025] The internal reward is defined as: for a pair of clean speech and noisy speech, let the clean speech signal be x', the noisy speech be y', and the intermediate speech be x mean , x mean Get the formula for x mean =tx′+(1-t)y′, where time step t is a number that changes from 0 to 1 during the training process. Here, the denoised speech obtained in the tth step of speech enhancement is x1 mean , in the tth step of the training process, the speech quality assessment model calculates x mean mos score scorel and x1 mean The mos score score2 is given. If score1 is greater than or equal to score2, a reward of 1 is given. If score1 is less than score2, a reward of -1 is given.

[0026] Optionally, the speech enhancement model to be fine-tuned is used as the Actor model, the Critic model parameters are initialized, the speech enhancement model to be fine-tuned is trained using the unsupervised speech data set and the Actor-Critic algorithm, the speech quality assessment model is used as feedback given by the environment, so that the speech enhancement model to be fine-tuned takes improving speech quality as the goal, and a reward of -1 is given for each step in the diffusion. In the process of using the trained speech enhancement model to be fine-tuned as the final speech enhancement model, the current reward r is obtained according to the defined external reward. t ; In the reward definition, even if there is no internal reward, a reward of -1 will still be given for each step.

[0027] In a second aspect, the present application provides a speech enhancement method based on an Actor-Critic algorithm and a diffusion model, wherein the speech enhancement method based on the Actor-Critic algorithm and the diffusion model comprises:

[0028] Acquire a speech evaluation data set and an unsupervised speech data set; the speech evaluation data set includes NISQA, PSTN, and Tencent data sets; the unsupervised speech data set includes different speech;

[0029] Using the speech evaluation data set to train a NISQA model, and using the trained NISQA model as a speech quality evaluation model;

[0030] The NCSN++ model used for diffusion is used as the Actor model, and a multi-layer Transformer stack is used, followed by an adaptive pooling layer as the Critic model. The NCSN++ model is trained using the unsupervised speech dataset and the Actor-Critic algorithm. The speech quality assessment model is used as feedback given by the environment to train the NCSN++ model to improve speech quality. For each step in the diffusion, a reward of -1 is given, and the trained NCSN++ model is used as the final speech enhancement model.

[0031] The final speech enhancement model is used to perform speech enhancement on the speech to be enhanced to obtain clean speech.

[0032] Optionally, using the speech evaluation data set to train a NISQA model, and using the trained NISQA model as a speech quality evaluation model, specifically includes:

[0033] The speech evaluation data set is used to train the NISQA model, the input of the NISQA model is speech data, the output of the NISQA model is a speech quality evaluation score, the loss function is the square difference between the score predicted by the NISQA model and the speech quality score in the speech evaluation data set, and the loss function is used to train the NISQA model until it converges to obtain a speech quality evaluation model.

[0034] Optionally, the NCSN++ model used for diffusion is used as an Actor model, and a multi-layer Transformer stack is used, followed by an adaptive pooling layer as a Critic model. The NCSN++ model is trained using the unsupervised speech dataset and the Actor-Critic algorithm, and the speech quality assessment model is used as feedback given by the environment to train the NCSN++ model to improve speech quality. For each step in the diffusion, a reward of -1 is given, and the trained NCSN++ model is used as the final speech enhancement model, which specifically includes:

[0035] Step P1: Initialize the maximum number of steps L and the Critic model parameters of the speech enhancement based on the diffusion model, and randomly initialize the NCSN++ model parameters;

[0036] Step P2: randomly extracting speech from the unsupervised speech data set, and using Fourier transform to obtain a noisy speech spectrum y;

[0037] Step P3: Initialize time step t to 0;

[0038] Step P4: Record the current noise speech spectrum y and time step t as environment s t , the environment s t Input into the NCSN++ model to obtain the predicted noise spectrum of the current output of the NCSN++ model, recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to get the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean is the final enhanced spectrum result, otherwise the intermediate result spectrum of the current NCSN++ model is defined as x t+1 / L =xt mean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 ;

[0039] Step P5: Get the current reward r according to the defined external reward t , the environment s t and t+1 and its corresponding actions, provided to the Critic model, using the function Calculate the Critic model loss function L w , use the reverse gradient algorithm to update the current Critic model parameters; where γ is the discount factor, is the output of the Critic model at time t, is the output of the Critic model at time t+1;

[0040] Step P6: Take the inverse of the output of the Critic model as the loss function of the NCSN++ model, and use the reverse gradient algorithm to update the NCSN++ model parameters;

[0041] Step P7: Repeat steps P4-P6 until t reaches 1, at which point one round of update is completed, and t = 1;

[0042] Step P8: Repeat steps P2-P7 and train until the NCSN++ model and the Critic model converge.

[0043] Optionally, the external reward is defined as follows: let L be the maximum number of steps of speech enhancement based on the diffusion model. After the diffusion model enhancement is denoised by L steps, the enhanced speech x0 is obtained. At this time, the NISQA model is used to score x0 to obtain a score score. If the score is greater than or equal to 4 at this time, the environment gives feedback 2*L. If the score is less than 4 at this time, a reward of -L is given. In order to encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each step in the diffusion.

[0044] Optionally, in the reward definition, even if there is no intrinsic reward, a reward of -1 is still given at each step.

[0045] According to the specific embodiments provided in this application, this application has the following technical effects:

[0046] The present application provides a speech enhancement method based on an Actor-Critic algorithm and a diffusion model. For the first time, a reinforcement learning algorithm, namely, the Actor-Critic algorithm, is used in a speech enhancement method based on a diffusion model. Compared with the existing speech enhancement method based on a diffusion model that uses a specific SDE equation for calculation, the present application uses the NCSN++ model used for diffusion as an Actor model, uses the Actor model to execute the diffusion process, uses a multi-layer Transformer stack, and a Critic model followed by an adaptive pooling layer to guide the Actor model training. By using the Actor-Critic algorithm to train the NCSN++ model, and using a speech quality assessment model as feedback given by the environment, the NCSN++ model training is aimed at improving speech quality without being restricted by a specific SDE equation, etc., further improving the generalization performance, and achieving a good speech enhancement effect in different scenarios. For each step in the diffusion, by giving a -1 reward, the speech enhancement model is urged to use as few diffusion steps as possible to achieve efficient speech enhancement, thereby achieving the effect of obtaining clean speech with good quality relatively quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 A flowchart of a speech enhancement method based on an Actor-Critic algorithm and a diffusion model provided in Example 1 of the present application;

[0049] Figure 2 A flowchart of a speech enhancement method based on an Actor-Critic algorithm and a diffusion model provided in Example 2 of the present application;

[0050] Figure 3 Schematic diagram of the relationship between the Actor model and the Critic model. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0052] The purpose of this application is to provide a speech enhancement method based on the Actor-Critic algorithm and the diffusion model, which can achieve a good speech enhancement effect in different scenarios and can also obtain clean speech with good quality relatively quickly.

[0053] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0054] Example 1

[0055] like Figure 1 As shown, a speech enhancement method based on an Actor-Critic algorithm and a diffusion model provided in Example 1 of the present application includes:

[0056] Step 101: Acquire a speech enhancement dataset, a speech evaluation dataset, and an unsupervised speech dataset; the speech enhancement dataset includes the VoiceBank-DEMAND dataset; the speech evaluation dataset includes the NISQA, PSTN, and Tencent datasets; and the unsupervised speech dataset includes different voices.

[0057] Step 102: Use the speech evaluation data set to train the NISQA model, and use the trained NISQA model as a speech quality evaluation model.

[0058] The step 102 specifically includes:

[0059] The NISQA model is trained using a speech evaluation dataset. The input of the NISQA model is speech data, and the output of the NISQA model is the speech quality evaluation score. The loss function is the square difference between the score predicted by the NISQA model and the speech quality score in the speech evaluation dataset. The loss function is used to train the NISQA model until it converges to obtain a speech quality evaluation model.

[0060] Step 103: The NCSN++ model used for diffusion is used as the Actor model. A multi-layer Transformer stack is used, followed by an adaptive pooling layer as the Critic model. The speech enhancement dataset and the Actor-Critic algorithm are used to train the NCSN++ model. The speech quality assessment model is used as feedback given by the environment to train the NCSN++ model to improve speech quality. For each step in the diffusion, a reward of -1 is given, and the trained NCSN++ model is used as the speech enhancement model to be fine-tuned.

[0061] The step 103 specifically includes:

[0062] Step S1: Initialize the maximum number of steps L for speech enhancement based on the diffusion model, and randomly initialize the NCSN++ model parameters and the Critic model parameters.

[0063] Step S2: Randomly extract clean speech and noisy speech from the speech enhancement dataset, and use Fourier transform to obtain the clean speech spectrum x and the noisy speech spectrum y.

[0064] Step S3: Initialize time step t to 0.

[0065] Step S4: Record the current noise speech spectrum y and time step t as environment s t , the environment s t Input into the NCSN++ model to obtain the predicted noise spectrum of the current output of the NCSN++ model, recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to get the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean is the final enhanced spectrum result, otherwise the intermediate result spectrum of the current NCSN++ model is defined as x t+1 / L =xt mean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 .

[0066] Step S5: Get the current reward r according to the defined external reward and internal rewardt , the environment s t and S t+1 and its corresponding actions, provided to the Critic model, using the function Calculate the Critic model loss function L w , use the reverse gradient algorithm to update the current Critic model parameters; where γ is the discount factor, is the output of the Critic model at time t, It is the output of the Critic model at the t+1th moment.

[0067] Step S6: The output of the Critic model is inverted as the loss function of the NCSN++ model, and the NCSN++ model parameters are updated using the reverse gradient algorithm.

[0068] Step S7: Repeat steps S4-S6 until t reaches 1, at which time one round of updating is completed, and t=1.

[0069] Step S8: Repeat steps S2-S7 and train until the NCSN++ model and the Critic model converge.

[0070] The external reward is defined as follows: Let L be the maximum number of steps for speech enhancement based on the diffusion model. After L steps of denoising, the enhanced speech x0 is obtained. At this time, the NISQA model is used to score x0 to obtain a score. If the score is greater than or equal to 4, the environment gives feedback 2*L. If the score is less than 4, a reward of -L is given. In order to encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each step in the diffusion.

[0071] The internal reward is defined as: For a pair of clean speech and noisy speech, let the clean speech signal be x', the noisy speech be y', and the intermediate speech be x mean , x mean Get the formula for x mean =tx′+(1-t)y′, where time step t is a number that changes from 0 to 1 during the training process. Here, the denoised speech obtained in the tth step of speech enhancement is x1 mean , in the tth step of the training process, the speech quality assessment model calculates x mean mos score scorel and x1 mean The mos score score2 is given. If score1 is greater than or equal to score2, a reward of 1 is given. If score1 is less than score2, a reward of -1 is given.

[0072] Step 104: The speech enhancement model to be fine-tuned is used as the Actor model, the Critic model parameters are initialized, the speech enhancement model to be fine-tuned is trained using the unsupervised speech dataset and the Actor-Critic algorithm, and the speech quality assessment model is used as the feedback given by the environment so that the speech enhancement model to be fine-tuned takes improving speech quality as the goal. For each step in the diffusion, a reward of -1 is given, and the trained speech enhancement model to be fine-tuned is used as the final speech enhancement model.

[0073] In the process of step 104, the current reward r is obtained according to the defined external reward. t ; In the reward definition, even if there is no internal reward, a reward of -1 will still be given for each step.

[0074] Step 105: Use the final speech enhancement model to perform speech enhancement on the speech to be enhanced to obtain clean speech.

[0075] The following is a specific example to illustrate the technical solution of the speech enhancement method based on the Actor-Critic algorithm and the diffusion model provided in Example 1 of the present application:

[0076] Step 1: Dataset collection, training data set: This part is divided into:

[0077] 1. Speech enhancement dataset: Collect data with clean speech and noisy speech pairs, which can be collected through public datasets on the Internet, such as the voicebank-demand dataset (i.e., VoiceBank-DEMAND dataset).

[0078] 2. Speech evaluation dataset: Collect non-embedded speech evaluation datasets, such as NISQA, PSTN, Tencent datasets, etc. These datasets have speech and corresponding human subjective scores of speech, namely mos scores.

[0079] 3. Unsupervised speech dataset: Collect speech data from the Internet, etc., without requiring a clean dataset.

[0080] Step 2: Train the speech quality assessment model.

[0081] The NISQA model is selected as the model and trained using the speech evaluation dataset collected in step 1. The model input is speech data and the output is the speech quality evaluation score. The loss function is the square difference between the predicted score and the speech quality score in the dataset, that is:

[0082] loss=||pred-target|| 2 ;

[0083] Among them, pred is the model output and target is the real speech quality assessment score.

[0084] The loss function is used to train the model until it converges to obtain the speech quality assessment model.

[0085] Step 3: Use the Actor-Critic algorithm to train the speech enhancement model. Before starting the training, define the following elements first:

[0086] Rewards: Rewards are divided into two parts, external rewards and internal rewards:

[0087] External reward: Let L be the maximum number of steps for speech enhancement based on the diffusion model. After L steps of denoising, the enhanced speech x0 is obtained. At this time, the NISQA model is used to score x0 to obtain the score score. If the score is greater than or equal to 4, the environment gives feedback 2*L. If the score is less than 4, a reward -L is given. In order to encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each step in the diffusion.

[0088] Internal reward: For a pair of clean speech and noisy speech, let the clean speech signal be x' and the noisy speech be y'. Get the intermediate speech as x mean , x mean The formula for obtaining is as follows:

[0089] x mean =tx'+(1-t)y';

[0090] Among them, t is the time step, which is a number that changes from 0 to 1 during the training process. Here, the denoised speech obtained in the tth step of speech enhancement is x1 mean , in the tth step of the training process, the NISQA calculation x obtained in step 2 is used mean and x1 mean The mos scores score1 and score2, if score1 is greater than or equal to score2, a reward of 1 is given, if score1<score2: a reward of -1 is given.

[0091] Model:

[0092] Actor model: The NCSN++ model is used as the speech enhancement model. The model inputs the time step t and the noisy speech spectrum (obtained by Fourier transforming the noisy speech) and outputs the predicted noise spectrum.

[0093] Critic model: A multi-layer transformer stack followed by an adaptive pooling layer is used as the model, denoted as V below. w, where w is the current critic model (i.e., Critic model) parameter. The input is the noisy speech at each time step (in the specific training step, the noisy speech at each time step can be obtained by inverse Fourier transforming the intermediate spectrum) and the output of the actor model (i.e., Actor model), and the output is the expected reward for the actor model to perform the action at the current time step (the expected reward is to score the action, which is a concept in reinforcement learning and is an overall evaluation of the entire denoising process).

[0094] The above contents about rewards and model definitions are all the process of definition, mainly to clarify some details in the following training process. The NISQA model exists as part of the reward in this application, and NCSN++ is defined as an actor model and critic in the above text. These are all for the needs of using the reinforcement learning algorithm actor-critic for reinforcement learning training. The final speech enhancement model only has the NCSN++ model, which is also an actor model.

[0095] The specific training process of the speech enhancement model with clean speech supervision (the speech enhancement model with clean speech supervision is only the actor model, and the actor model and the NCSN++ model here are the same, except that in the reinforcement learning framework, that is, the actor-critic framework, the model is called the actor model, but the specific implementation of the model is the NCSN++ model) is as follows:

[0096] (1) Initialize the maximum number of steps L for speech enhancement based on the diffusion model, and randomly initialize the speech enhancement model parameters and the critic model parameters.

[0097] (2) Select the speech enhancement dataset in step 1, randomly extract clean speech and noisy speech from the dataset (i.e., randomly extract any pair of clean speech and noisy speech), and use Fourier transform to obtain the clean speech spectrum x and the noisy speech spectrum y.

[0098] (3) Initialize time step t to 0.

[0099] (4) The current noise speech spectrum y and time step t are recorded as environment s t , the environment s t Input it into the Actor model defined in the above step to obtain the predicted noise spectrum of the Actor's current output, recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to get the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean is the final enhanced spectrum result, otherwise the intermediate result spectrum of the current speech enhancement model is defined as x t+1 / L =xtmean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 (By passing the environment s t+1 Input into the NCSN++ model and get the action a output by the NCSN++ model t+1 ). Where t is the current time step, 1 / L represents the step size, and t+1 / L is the updated time step.

[0100] (5) Based on the external rewards and internal rewards defined above, obtain the current reward r t , the environment s t and t+1 and its corresponding actions, provided to the critic, and the following function is used to calculate the critic model loss function L w :

[0101]

[0102] Use the reverse gradient algorithm to update the current critic model parameters. Among them, γ is the discount factor used to weigh the immediate reward and long-term reward, which is set to 0.9. is the output of the Critic model at time t, that is, the score at time t, It is the output of the Critic model at time t+1, that is, the score at time t+1.

[0103] (6) Calculate the loss function of the actor model: The output of the critic model Take the inverse as the loss function L of the actor model actor ,Right now:

[0104]

[0105] Update the actor model parameters using the back-gradient algorithm.

[0106] (7) Repeat steps (4)-(6) until t reaches 1, at which point one round of update is completed, and t=1.

[0107] (8) Repeat steps (2)-(7) until the model (i.e., actor model and critic model) converges. Convergence condition: If the loss function value does not decrease significantly after multiple consecutive iterations, the model can be considered to have converged.

[0108] Fine-tuning steps:

[0109] For the speech enhancement model without clean speech supervision (the model composition of the speech enhancement model without clean speech supervision is the same as that of the speech enhancement model with clean speech supervision, but there is a lack of internal rewards during training), the training is as follows:

[0110] (1) Initialize the maximum number of steps L of speech enhancement based on the diffusion model and the critic model parameters. If the model is fine-tuned, the parameters of the speech enhancement model (NCSN++\actor model) are initialized to the model parameters trained with the above supervised dataset.

[0111] (2) Randomly extract speech from the unsupervised speech dataset and use Fourier transform to obtain the noisy speech spectrum y.

[0112] (3) Initialize time step t to 0.

[0113] (4) The current noise speech spectrum y and time step t are recorded as environment s t , the environment s t Input it into the Actor model defined in the above step to obtain the predicted noise spectrum of the Actor's current output, recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to get the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean is the final enhanced spectrum result, otherwise the intermediate result spectrum of the current speech enhancement model is defined as x t+1 / L =xt mean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 .

[0114] (5) According to the external reward defined above (in the reward definition, even if there is no internal reward, a reward of -1 will still be given for each step, which is to encourage the use of fewer steps), obtain the current reward r t , the environment s t and t+1 and its corresponding actions, provided to the critic, and the following function is used to calculate the critic model loss function L w :

[0115]

[0116] Use the reverse gradient algorithm to update the current critic model parameters. Where γ is the discount factor, is the output of the Critic model at time t, It is the output of the Critic model at the t+1th moment.

[0117] (6) Calculate the loss function of the actor model: The output of the critic model Take the inverse as the loss function L of the actor model actor ,Right now:

[0118]

[0119] Update the actor model parameters using the back-gradient algorithm.

[0120] (7) Repeat steps (4)-(6) until t reaches 1, at which point one round of update is completed, and t=1.

[0121] (8) Repeat steps (2)-(7) and train until the model (i.e., the critic and actor models) converges. Convergence condition: If the loss function value does not decrease significantly after multiple consecutive iterations, the model can be considered to have converged.

[0122] Subsequent use:

[0123] In the clean speech supervision, a preliminary speech enhancement model will be trained, and unsupervised data will be used for speech enhancement training to enrich the training data set. At the same time, since clean speech supervision is not required, this method is relatively cheap. For various scenarios, after defining the number of diffusion steps, the noisy audio and time step t (initial 0) can be directly provided to the speech enhancement model for iterative denoising. The specific steps are as (3)-(4) in the training process in step 3 above to achieve the purpose of speech enhancement. When the model performs poorly in an unknown field, the above-mentioned unsupervised speech enhancement training method (i.e., the speech enhancement model training process without clean speech supervision) can be used to fine-tune the model (i.e., fine-tune the actor model parameters) without the need for clean speech.

[0124] The training of the speech enhancement method based on the Actor-Critic algorithm and the diffusion model provided in Example 1 of the present application starts with using clean and noisy speech pairs, but the more important part is actually the unsupervised data set. This method expands the data set for training the speech enhancement model without increasing the cost too much.

[0125] Example 2

[0126] like Figure 2 As shown, a speech enhancement method based on an Actor-Critic algorithm and a diffusion model provided in Example 2 of the present application includes:

[0127] Step 201: Acquire a speech evaluation dataset and an unsupervised speech dataset; the speech evaluation dataset includes NISQA, PSTN, and Tencent datasets; the unsupervised speech dataset includes different speech.

[0128] Step 202: Use the speech evaluation data set to train the NISQA model, and use the trained NISQA model as a speech quality evaluation model.

[0129] The step 202 specifically includes:

[0130] The NISQA model is trained using a speech evaluation dataset. The input of the NISQA model is speech data, and the output of the NISQA model is the speech quality evaluation score. The loss function is the square difference between the score predicted by the NISQA model and the speech quality score in the speech evaluation dataset. The loss function is used to train the NISQA model until it converges to obtain a speech quality evaluation model.

[0131] Step 203: The NCSN++ model used for diffusion is used as the Actor model, and a multi-layer Transformer stack is used, followed by an adaptive pooling layer as the Critic model. The NCSN++ model is trained using an unsupervised speech dataset and the Actor-Critic algorithm. The speech quality assessment model is used as feedback given by the environment, so that the NCSN++ model is trained to improve speech quality. For each step in the diffusion, a reward of -1 is given, and the trained NCSN++ model is used as the final speech enhancement model.

[0132] The step 203 specifically includes:

[0133] Step P1: Initialize the maximum number of steps L and the Critic model parameters of the diffusion model-based speech enhancement, and randomly initialize the NCSN++ model parameters.

[0134] Step P2: Randomly extract speech from the unsupervised speech dataset and use Fourier transform to obtain the noisy speech spectrum y.

[0135] Step P3: Initialize time step t to 0.

[0136] Step P4: Record the current noise speech spectrum y and time step t as environment s t , the environment s t Input into the NCSN++ model to obtain the predicted noise spectrum of the current output of the NCSN++ model, recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to get the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt meanis the final enhanced spectrum result, otherwise the intermediate result spectrum of the current NCSN++ model is defined as x t+1 / L =xt mean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 .

[0137] Step P5: Get the current reward r according to the defined external reward t , the environment s t and t+1 and its corresponding actions, provided to the Critic model, using the function Calculate the Critic model loss function L w , use the reverse gradient algorithm to update the current Critic model parameters; where γ is the discount factor, is the output of the Critic model at time t, It is the output of the Critic model at the t+1th moment.

[0138] The external reward is defined as follows: Let L be the maximum number of steps for speech enhancement based on the diffusion model. After L steps of denoising, the enhanced speech x0 is obtained. At this time, the NISQA model is used to score x0 to obtain a score. If the score is greater than or equal to 4, the environment gives feedback 2*L. If the score is less than 4, a reward of -L is given. In order to encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each step in the diffusion.

[0139] In the reward definition, even if there is no internal reward, a reward of -1 is still given for each step.

[0140] Step P6: Invert the output of the Critic model as the loss function of the NCSN++ model, and use the reverse gradient algorithm to update the NCSN++ model parameters.

[0141] Step P7: Repeat steps P4-P6 until t reaches 1, at which time one round of update is completed, and t=1.

[0142] Step P8: Repeat steps P2-P7 and train until the NCSN++ model and the Critic model converge.

[0143] Step 204: Use the final speech enhancement model to perform speech enhancement on the speech to be enhanced to obtain clean speech.

[0144] The following is a specific example to illustrate the technical solution of the speech enhancement method based on the Actor-Critic algorithm and the diffusion model provided in Example 2 of the present application:

[0145] Step 1: Dataset collection, training data set: This part is divided into:

[0146] 1. Speech evaluation dataset: Collect non-embedded speech evaluation datasets, such as NISQA, PSTN, Tencent datasets, etc. These datasets have speech and corresponding human subjective scores of speech, namely mos scores.

[0147] 2. Unsupervised speech dataset: Collect speech data from the Internet, etc., without requiring a clean dataset.

[0148] Step 2: Train the speech quality assessment model.

[0149] The NISQA model is selected as the model and trained using the speech evaluation dataset collected in step 1. The model input is speech data and the output is the speech quality evaluation score. The loss function is the square difference between the predicted score and the speech quality score in the dataset, that is:

[0150] loss=||pred-target|| 2 ;

[0151] Among them, pred is the model output and target is the real speech quality assessment score.

[0152] The loss function is used to train the model until it converges to obtain the speech quality assessment model.

[0153] Step 3: Use the Actor-Critic algorithm to train the speech enhancement model. Before starting the training, define the following elements first:

[0154] award:

[0155] External reward: Let L be the maximum number of steps for speech enhancement based on the diffusion model. After L steps of denoising, the enhanced speech x0 is obtained. At this time, the NISQA model is used to score x0 to obtain the score score. If the score is greater than or equal to 4, the environment gives feedback 2*L. If the score is less than 4, a reward -L is given. In order to encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each step in the diffusion.

[0156] Model:

[0157] Actor model: The NCSN++ model is used as the speech enhancement model. The model inputs the time step t and the noisy speech spectrum (obtained by Fourier transforming the noisy speech) and outputs the predicted noise spectrum.

[0158] Critic model: A multi-layer transformer stack followed by an adaptive pooling layer is used as the model, denoted as V below. w , where w is the current critic model (i.e., Critic model) parameter. The input is the noisy speech at each time step (in the specific training step, the noisy speech at each time step can be obtained by inverse Fourier transforming the intermediate spectrum) and the output of the actor model (i.e., Actor model), and the output is the expected reward for the actor model to perform the action at the current time step (the expected reward is to score the action, which is a concept in reinforcement learning and is an overall evaluation of the entire denoising process).

[0159] The above contents about rewards and model definitions are all the process of definition, mainly to clarify some details in the following training process. The NISQA model exists as part of the reward in this application, and NCSN++ is defined as an actor model and critic in the above text. These are all for the needs of using the reinforcement learning algorithm actor-critic for reinforcement learning training. The final speech enhancement model only has the NCSN++ model, which is also an actor model.

[0160] For a speech enhancement model without clean speech supervision (different from the speech enhancement method based on the Actor-Critic algorithm and the diffusion model provided in Example 1, the speech enhancement method based on the Actor-Critic algorithm and the diffusion model provided in Example 2 may not use a supervised data set, i.e., a speech enhancement data set, but directly use an unsupervised data set to train the speech enhancement model without clean speech supervision) training is as follows:

[0161] (1) Initialize the maximum number of steps L and critic model parameters of the diffusion model-based speech enhancement. If the speech enhancement model is directly trained without clean speech supervision, the speech enhancement model parameters are randomly initialized.

[0162] (2) Randomly extract speech from the unsupervised speech dataset and use Fourier transform to obtain the noisy speech spectrum y.

[0163] (3) Initialize time step t to 0.

[0164] (4) The current noise speech spectrum y and time step t are recorded as environment s t , the environment s tInput it into the Actor model defined in the above step to obtain the predicted noise spectrum of the Actor's current output, recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to get the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean is the final enhanced spectrum result, otherwise the intermediate result spectrum of the current speech enhancement model is defined as x t+1 / L =xt mean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 (By passing the environment s t+1 Input into the NCSN++ model and get the action a output by the NCSN++ model t+1 ). Where t is the current time step, 1 / L represents the step size, and t+1 / L is the updated time step.

[0165] (5) According to the external reward defined above (in the reward definition, even if there is no internal reward, a reward of -1 will still be given for each step, which is to encourage the use of fewer steps), obtain the current reward r t , the environment s t and t+1 and its corresponding actions, provided to the critic, and the following function is used to calculate the critic model loss function L w :

[0166]

[0167] Use the reverse gradient algorithm to update the current critic model parameters. Among them, γ is the discount factor used to weigh the immediate reward and long-term reward, which is set to 0.9. is the output of the Critic model at time t, that is, the score at time t, It is the output of the Critic model at time t+1, that is, the score at time t+1.

[0168] (6) Calculate the loss function of the actor model: The output of the critic model Take the inverse as the loss function L of the actor model actor ,Right now:

[0169]

[0170] Update the actor model parameters using the back-gradient algorithm.

[0171] (7) Repeat steps (4)-(6) until t reaches 1, at which point one round of update is completed, and t=1.

[0172] (8) Repeat steps (2)-(7) and train until the model (i.e., the critic and actor models) converges. Convergence condition: If the loss function value does not decrease significantly after multiple consecutive iterations, the model can be considered to have converged.

[0173] This application is a combination of diffusion model and reinforcement learning. Reinforcement learning uses the actor-critic framework. The relationship between the actor model and the critic model is as follows: Figure 3 As shown, the actor model is an enhanced model, which is specifically implemented as the NCSN++ model. The diffusion model is not a specific model in the method of this application, but a method, a method with certain generalization performance, and this application further improves its generalization performance.

[0174] At present, most of the research on speech enhancement focuses on processing the speech signal of a single speaker, which is usually pure speech recorded in a laboratory environment, and is more inclined to theoretical research. However, in practical applications, speech signals are often affected by various noises and interferences, such as background noise, echo, reverberation, etc. This puts forward requirements for the generalization performance of existing speech enhancement models. The diffusion model solves this problem to a certain extent, but there are still the following problems: 1. The existing diffusion model relies on pre-defined SDE equations, etc. These assumptions limit the generalization performance of the model score to a certain extent. 2. When giving the speech enhancement of the diffusion model, multiple rounds of iterative denoising are required during reasoning, and the time consumption is too large during reasoning. To solve the above problems, this application adopts the method of reinforcement learning (Actor-sritic algorithm is an algorithm of reinforcement learning) to train the model, which does not rely on a specific SDE equation. At the same time, because the research is based on the diffusion model, the model trained by this application retains the generalization performance of the diffusion model. At the same time, because the reward function is used to encourage the speech enhancement model to use fewer diffusion steps during training, the trained speech enhancement model can also reduce the number of iterations and achieve rapid speech enhancement.

[0175] Compared with the existing speech enhancement methods, the present application is also a speech enhancement method based on the diffusion model, but it is improved on the basis of the existing speech enhancement method based on the diffusion model, that is, the reinforcement learning algorithm actor-critic algorithm is combined with the diffusion model, which avoids the need for speech pairs to a certain extent and does not require the assumption of the SDE equation.

[0176] Compared with existing speech enhancement methods, this application does not rely on various assumptions such as pre-defined SDE formulas, and has high generalization and improved reasoning speed, which is mainly due to:

[0177] 1. The actor-critic algorithm is used to train the model, and the speech quality assessment model is used as the feedback given by the environment, so that the model training aims to improve the speech quality without being restricted by the SDE equation.

[0178] 2. Model training is performed based on a diffusion model (i.e., a diffusion model with multiple noise addition and denoising methods is used), retaining the generalization performance of the diffusion model itself (the diffusion model itself has generalization performance, but when it relies on clean noise speech pairs, the generalization performance is still a bit limited. This application uses a reinforcement learning algorithm, the actor-critc algorithm, to further improve the generalization). At the same time, the model can be trained without clean speech supervision, which greatly reduces the cost and price of migrating the model to unknown areas.

[0179] 3. A reward of -1 is given for each diffusion step in the reward to urge the speech enhancement model to use as few diffusion steps as possible to achieve efficient speech enhancement.

[0180] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A speech enhancement method based on Actor-Critic algorithm and diffusion model, characterized in that: The speech enhancement method based on the Actor-Critic algorithm and the diffusion model includes: Acquire a speech enhancement dataset, a speech evaluation dataset, and an unsupervised speech dataset; the speech enhancement dataset includes a VoiceBank-DEMAND dataset; the speech evaluation dataset includes a NISQA, PSTN, and Tencent dataset; and the unsupervised speech dataset includes different voices; Using the speech evaluation data set to train a NISQA model, and using the trained NISQA model as a speech quality evaluation model; The NCSN++ model used for diffusion is used as the Actor model, and a multi-layer Transformer stack is used, followed by an adaptive pooling layer as the Critic model. The NCSN++ model is trained using the speech enhancement dataset and the Actor-Critic algorithm. The speech quality assessment model is used as feedback given by the environment to train the NCSN++ model with the goal of improving speech quality. For each step in the diffusion, a reward of -1 is given, and the trained NCSN++ model is used as the speech enhancement model to be fine-tuned. The speech enhancement model to be fine-tuned is used as the Actor model, the Critic model parameters are initialized, the speech enhancement model to be fine-tuned is trained using the unsupervised speech data set and the Actor-Critic algorithm, the speech quality assessment model is used as feedback given by the environment, so that the speech enhancement model to be fine-tuned takes improving speech quality as the goal, a reward of -1 is given for each step in the diffusion, and the trained speech enhancement model to be fine-tuned is used as the final speech enhancement model; The final speech enhancement model is used to perform speech enhancement on the speech to be enhanced to obtain clean speech.

2. The method for speech enhancement based on Actor-Critic algorithm and diffusion model according to claim 1, characterized in that: The NISQA model is trained using the speech evaluation data set, and the trained NISQA model is used as a speech quality evaluation model, specifically including: The speech evaluation data set is used to train the NISQA model, the input of the NISQA model is speech data, the output of the NISQA model is a speech quality evaluation score, the loss function is the square difference between the score predicted by the NISQA model and the speech quality score in the speech evaluation data set, and the loss function is used to train the NISQA model until it converges to obtain a speech quality evaluation model.

3. The speech enhancement method based on Actor-Critic algorithm and diffusion model according to claim 1, characterized in that: The NCSN++ model used for diffusion is used as the Actor model, and a multi-layer Transformer stack is used, followed by an adaptive pooling layer as the Critic model. The NCSN++ model is trained using the speech enhancement dataset and the Actor-Critic algorithm. The speech quality assessment model is used as feedback given by the environment to train the NCSN++ model to improve speech quality. For each step in the diffusion, a reward of -1 is given, and the trained NCSN++ model is used as the speech enhancement model to be fine-tuned, specifically including: Step S1: Initialize the maximum number of steps L of speech enhancement based on the diffusion model, and randomly initialize the NCSN++ model parameters and the Critic model parameters; Step S2: randomly extracting clean speech and noisy speech from the speech enhancement data set, and using Fourier transform to obtain a clean speech spectrum x and a noisy speech spectrum y; Step S3: Initialize time step t to 0; Step S4: Record the current noise speech spectrum y and time step t as environment s t , the environment s t Input into the NCSN++ model to obtain the predicted noise spectrum of the current output of the NCSN++ model, recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to get the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean is the final enhanced spectrum result, otherwise the intermediate result spectrum of the current NCSN++ model is defined as x t+1 / L =xt mean +noiuse, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 ; Step S5: Get the current reward r according to the defined external reward and internal reward t , the environment s t and t+1 and its corresponding actions, provided to the Critic model, using the function Calculate the Critic model loss function L w , use the reverse gradient algorithm to update the current Critic model parameters; where γ is the discount factor, is the output of the Critic model at time t, is the output of the Critic model at time t+1; Step S6: The output of the Critic model is inverted as the loss function of the NCSN++ model, and the NCSN++ model parameters are updated using the reverse gradient algorithm; Step S7: Repeat steps S4-S6 until t reaches 1, at which point one round of update is completed, and t=1; Step S8: Repeat steps S2-S7 and train until the NCSN++ model and the Critic model converge.

4. The method for speech enhancement based on Actor-Critic algorithm and diffusion model according to claim 3, characterized in that: The external reward is defined as follows: Let L be the maximum number of steps of speech enhancement based on the diffusion model. After L steps of denoising, the enhanced speech x0 is obtained. At this time, the NISQA model is used to score x0 to obtain a score. If the score is greater than or equal to 4, the environment gives feedback 2*L. If the score is less than 4, a reward of -L is given. In order to encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each step in the diffusion. The internal reward is defined as: for a pair of clean speech and noisy speech, let the clean speech signal be x', the noisy speech be y', and the intermediate speech be x mean , x mean Get the formula for x mean =tx'+(1-t)y', where time step t is a number that changes from 0 to 1 during the training process. Here, the denoised speech obtained in the tth step of speech enhancement is x1 mean , in the tth step of the training process, the speech quality assessment model calculates x mean mos score score1 and x1 mean The mos score score2 is given. If score1 is greater than or equal to score2, a reward of 1 is given. If score1 is less than score2, a reward of -1 is given.

5. The method for speech enhancement based on Actor-Critic algorithm and diffusion model according to claim 4, characterized in that: The speech enhancement model to be fine-tuned is used as the Actor model, the Critic model parameters are initialized, the speech enhancement model to be fine-tuned is trained using the unsupervised speech data set and the Actor-Critic algorithm, the speech quality assessment model is used as the feedback given by the environment, so that the speech enhancement model to be fine-tuned takes improving speech quality as the goal, and a reward of -1 is given for each step in the diffusion. In the process of using the trained speech enhancement model to be fine-tuned as the final speech enhancement model, the current reward r is obtained according to the defined external reward. t ; In the reward definition, even if there is no internal reward, a reward of -1 will still be given for each step.

6. A speech enhancement method based on Actor-Critic algorithm and diffusion model, characterized in that: The speech enhancement method based on the Actor-Critic algorithm and the diffusion model includes: Acquire a speech evaluation data set and an unsupervised speech data set; the speech evaluation data set includes NISQA, PSTN, and Tencent data sets; the unsupervised speech data set includes different speech; Using the speech evaluation data set to train a NISQA model, and using the trained NISQA model as a speech quality evaluation model; The NCSN++ model used for diffusion is used as the Actor model, and a multi-layer Transformer stack is used, followed by an adaptive pooling layer as the Critic model. The NCSN++ model is trained using the unsupervised speech dataset and the Actor-Critic algorithm. The speech quality assessment model is used as feedback given by the environment to train the NCSN++ model to improve speech quality. For each step in the diffusion, a reward of -1 is given, and the trained NCSN++ model is used as the final speech enhancement model. The final speech enhancement model is used to perform speech enhancement on the speech to be enhanced to obtain clean speech.

7. The method for speech enhancement based on Actor-Critic algorithm and diffusion model according to claim 6, characterized in that: The NISQA model is trained using the speech evaluation data set, and the trained NISQA model is used as a speech quality evaluation model, specifically including: The speech evaluation data set is used to train the NISQA model, the input of the NISQA model is speech data, the output of the NISQA model is a speech quality evaluation score, the loss function is the square difference between the score predicted by the NISQA model and the speech quality score in the speech evaluation data set, and the loss function is used to train the NISQA model until it converges to obtain a speech quality evaluation model.

8. The method for speech enhancement based on Actor-Critic algorithm and diffusion model according to claim 6, characterized in that: The NCSN++ model used for diffusion is used as the Actor model, and a multi-layer Transformer stack is used, followed by an adaptive pooling layer as the Critic model. The NCSN++ model is trained using the unsupervised speech dataset and the Actor-Critic algorithm. The speech quality assessment model is used as feedback given by the environment to train the NCSN++ model to improve speech quality. For each step in the diffusion, a reward of -1 is given, and the trained NCSN++ model is used as the final speech enhancement model, which specifically includes: Step P1: Initialize the maximum number of steps L and the Critic model parameters of the speech enhancement based on the diffusion model, and randomly initialize the NCSN++ model parameters; Step P2: randomly extracting speech from the unsupervised speech data set, and using Fourier transform to obtain a noisy speech spectrum y; Step P3: Initialize time step t to 0; Step P4: Record the current noise speech spectrum y and time step t as environment s t , the environment s t Input into the NCSN++ model to obtain the predicted noise spectrum of the current output of the NCSN++ model, recorded as action a t , subtract the predicted noise spectrum from the noise speech spectrum y to get the intermediate noise spectrum xt mean , if t+1 / L=1 at this time, then xt mean is the final enhanced spectrum result, otherwise the intermediate result spectrum of the current NCSN++ model is defined as x t+1 / L =xt mean +noise, noise is a randomly generated standard Gaussian noise, at this time the intermediate result spectrum x t+1 / L and time step t+1 / L is recorded as environment s t+1 , and get action a t+1 ; Step P5: Get the current reward r according to the defined external reward t , the environment s t and t+1 and its corresponding actions, provided to the Critic model, using the function Calculate the Critic model loss function L w , use the reverse gradient algorithm to update the current Critic model parameters; where γ is the discount factor, is the output of the Critic model at time t, is the output of the Critic model at time t+1; Step P6: Take the inverse of the output of the Critic model as the loss function of the NCSN++ model, and use the reverse gradient algorithm to update the NCSN++ model parameters; Step P7: Repeat steps P4-P6 until t reaches 1, at which point one round of update is completed, and t = 1; Step P8: Repeat steps P2-P7 and train until the NCSN++ model and the Critic model converge.

9. The method for speech enhancement based on Actor-Critic algorithm and diffusion model according to claim 8, characterized in that: The external reward is defined as follows: Let L be the maximum number of steps of speech enhancement based on the diffusion model. After L steps of denoising, the enhanced speech x0 is obtained. At this time, the NISQA model is used to score x0 to obtain a score. If the score is greater than or equal to 4, the environment gives feedback 2*L. If the score is less than 4, a reward of -L is given. In order to encourage the diffusion model to use fewer steps for speech enhancement, a reward of -1 is given for each step in the diffusion.

10. The method for speech enhancement based on Actor-Critic algorithm and diffusion model according to claim 9, characterized in that: In the reward definition, even if there is no internal reward, a reward of -1 is still given for each step.

Images