Speech enhancement method and system based on deep learning

Abstract

The invention discloses a speech enhancement method and system based on deep learning. The method comprises the following steps: acquiring an FFT feature corresponding to original speech data; performing feature extraction on the FFT features based on a convolutional neural network to obtain convolutional feature information; processing the convolutional feature information based on a recurrent neural network to obtain processed convolutional feature information; based on different levels of feature fusion structures, weight coefficients of coding features and decoding features are obtained in combination with a convolutional neural network, and feature fusion information is obtained; performing feature extraction on the feature fusion information and the processed convolutional feature information based on a convolutional neural network to obtain deconvolution feature information; and solving multi-frame mask features based on the deconvolution feature information, and solving an output result of the current frame in combination with multi-frame input features. According to the method, the structural information of the voice signal can be effectively reserved through channel calculation based on the convolutional neural network and the recurrent neural network, and the purpose of reducing the overall parameter quantity of the model is achieved through a parameter sharing mode.

Description

Technical Field

[0001] The present invention relates to the technical field of audio information processing, and in particular to a speech enhancement method and system based on deep learning. Background Art

[0002] In traditional single-channel speech enhancement methods, some assumptions are usually made about the interaction between noise signals and clean speech signals. These assumptions are often based on specific mathematical models or statistical characteristics, which limits the performance and scope of application of enhancement techniques, resulting in unsatisfactory speech quality after enhancement. In contrast, deep learning-based methods do not require explicit assumptions about the relationship between speech signals and noise signals. This type of method uses a large amount of training data to directly learn the mapping relationship between clean speech and noise. The advantage of this method is that it can automatically extract features from data and learn complex nonlinear relationships, thereby overcoming some inherent problems of traditional speech enhancement algorithms and achieving better denoising effects and stronger generalization capabilities.

[0003] But it also brings new challenges. Deep learning algorithms often require a large number of parameters and computational overhead. The computational overhead problem in embedded devices has been solved to a certain extent with the development of chip performance. The size of model parameters often restricts the widespread application of deep learning models in devices, and the number of parameters is often closely related to the performance of the model.

[0004] Therefore, it is urgent to propose a speech enhancement method to solve the problem raised. Summary of the invention

[0005] Based on this, it is necessary to provide a speech enhancement method and system based on deep learning to address the shortcomings of the existing technology, so as to reduce the overall number of model parameters through parameter sharing.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] In a first aspect, a speech enhancement method based on deep learning is provided, which comprises the following steps:

[0008] Get the FFT features corresponding to the original speech data;

[0009] Extract FFT features based on convolutional neural network to obtain convolution feature information;

[0010] Processing the convolution feature information based on a recurrent neural network to obtain the processed convolution feature information;

[0011] Based on the feature fusion structure of different levels combined with the convolutional neural network, the weight coefficients of the encoding features and decoding features are obtained, and the encoding features and decoding features are multiplied by the corresponding weight coefficients respectively and then added to obtain the feature fusion information;

[0012] Based on the convolutional neural network, feature fusion information and processed convolution feature information are extracted to obtain deconvolution feature information;

[0013] The multi-frame mask features are obtained based on the deconvolution feature information, and the output result of the current frame is obtained by combining the multi-frame input features.

[0014] In one embodiment, before the step of obtaining the FFT features corresponding to the original speech data, the step further includes:

[0015] Build a speech enhancement model.

[0016] In one embodiment, the step of extracting features from FFT features based on a convolutional neural network to obtain convolution feature information comprises the following steps:

[0017] The convolution module has a total of five layers. The number of channels in the first three layers is set to 16, and the last two layers are grouped convolutions. The number of groups is equal to the number of input channels, which is 16. The first three layers of convolution are used to perform preliminary feature extraction on the FFT features, and then the last two layers of convolution are used to obtain the convolution feature information.

[0018] In one embodiment, the step of processing the convolution feature information based on a recurrent neural network to obtain the processed convolution feature information includes the following steps:

[0019] Define the input feature of LSTM on each channel as X, inter is the set of each LSTM input feature concatenated in the channel dimension, concat represents the concatenation operation, and the specific calculation formula satisfies:

[0020]

[0021] The concatenated set of LSTM input features in the channel dimension is normalized after calculating the mean and variance in the feature and channel dimensions to obtain the processed convolution feature information; where:

[0022] Mean

[0023] variance

[0024] The processed convolution feature information Y satisfies the following formula:

[0025]

[0026] Among them, ξ is a constant, λ and β are trainable parameters, and c, f, and X are defined as the number of channels, the number of features, and the LSTM input features, respectively.

[0027] In one embodiment, the steps are based on different levels of feature fusion structures combined with convolutional neural networks to obtain weight coefficients of encoding features and decoding features, and the encoding features and decoding features are multiplied by the corresponding weight coefficients and then added to obtain feature fusion information. The specific operations include:

[0028] The encoding feature ef and the decoding feature df are concatenated and input into a two-layer convolution with a kernel of 1*1 to obtain the mask features of encoding and decoding;

[0029] The obtained encoded and decoded mask features are mapped to [0, 1] through the Sigmoid function to obtain the weight coefficient corresponding to the encoded feature;

[0030] Multiply the encoding features and decoding features by the corresponding weight coefficients respectively and then add them together to obtain the feature fusion information e n =(Sigmoid(k(ef,df)))*ef+(1-Sigmoid(k(ef,df)))*df.

[0031] In one embodiment, the step performs feature extraction on the feature fusion information and the processed convolution feature information based on a convolutional neural network to obtain a method for deconvolution feature information, and the specific operations include:

[0032] The structure of the deconvolution module is symmetrical with that of the convolution module. The deconvolution module has five layers in total. The first two layers are group convolutions, and the last three layers are deconvolution layers that are symmetrically set with the convolution layers of the convolution module. Preliminary feature extraction is performed through the first two layers of group convolution, and then the last three layers of deconvolution are used to obtain deconvolution feature information.

[0033] In one embodiment, the step is to obtain the mask features of multiple frames based on the deconvolution feature information, and to obtain the output result of the current frame by combining the input features of multiple frames. The specific operations include:

[0034] The deconvolution output feature is represented by E, and the mask feature mask is represented by m. The relationship between the mask feature m and the deconvolution output feature E satisfies:

[0035] m = tanh(E);

[0036] Concatenate the input of the current frame with the input of the previous frame and the input of the next frame to obtain the input feature x of the current frame;

[0037] The input feature x of the current frame is dot-multiplied with the output mask feature m to obtain the output result of the current frame.

[0038] In one embodiment, the step of performing a dot product summation of the input feature x of the current frame and the output mask feature m to obtain the output result of the current frame includes:

[0039] The input feature x of the current frame is summed by the dot multiplication of the output mask feature m through deep filtering. The calculation expression is as follows: Among them, x(ct) represents the input features of the corresponding frame, m(ct) represents the multi-frame mask features, and ct represents the number of observed frames.

[0040] In a second aspect, a speech enhancement system based on deep learning is provided, which includes

[0041] FFT module, used to obtain FFT features corresponding to original speech data;

[0042] The convolution module is used to extract the FFT features based on the convolutional neural network and obtain the convolution feature information;

[0043] The RNN module is used to process the convolution feature information based on the recurrent neural network and obtain the processed convolution feature information;

[0044] The FFA module is used to obtain the weight coefficients of the encoding features and decoding features based on the different-level feature fusion (FFA) structures combined with the convolutional neural network, and multiply the encoding features and decoding features with the corresponding weight coefficients respectively and then add them to obtain feature fusion information;

[0045] A deconvolution module is used to extract features from feature fusion information and processed convolution feature information based on a convolutional neural network to obtain deconvolution feature information;

[0046] The deep filtering module is used to obtain multi-frame mask features based on the deconvolution feature information and obtain the output result of the current frame by combining the multi-frame input features.

[0047] In one embodiment, a construction module is also included for constructing a speech enhancement model.

[0048] In summary, the speech enhancement method and system based on deep learning of the present invention can effectively retain the structural information of the speech signal by calculating based on channels based on convolutional neural networks and recurrent neural networks. At the same time, the channel feature normalization method can make the model training more stable and reduce the impact of the output speech amplitude. Since the recurrent neural network module parameters account for the main part of the overall model parameters, the LSTM unit uses parameter sharing to achieve the purpose of reducing the overall model parameter amount, and the attention mechanism is used for the encoding and decoding information to make the model more effectively focus on the important information part. In addition, the deep filtering method is used to link the previous and next speech frame information, thereby further improving the output voice signal quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flowchart of a speech enhancement method based on deep learning provided by an embodiment of the present invention;

[0050] Figure 2 is a flowchart of step S120 provided in an embodiment of the present invention;

[0051] Figure 3 is a schematic diagram of the flow of the group convolution operation in step S120 provided in an embodiment of the present invention;

[0052] Figure 4 is a flowchart of step S130 provided in an embodiment of the present invention;

[0053] Figure 5 is a flowchart schematic diagram of step S140 provided in an embodiment of the present invention;

[0054] Figure 6 is a flowchart of step S150 provided in an embodiment of the present invention;

[0055] Figure 7 is a structural block diagram of a deep learning-based speech enhancement system provided by an embodiment of the present invention;

[0056] Figure 8 It is a structural principle diagram of a speech enhancement model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] It should be noted that the terms "first", "second", "third", "fourth", etc. in the specification and claims of the present invention are used to distinguish different objects rather than to describe a specific order. The terms "including" and "having" in the embodiments of the present invention and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0059] like Figure 1is a flow chart of a speech enhancement method based on deep learning provided by an embodiment of the present invention, such as Figure 1 As shown, the deep learning-based speech enhancement method includes steps S110 to S160, which are specifically as follows:

[0060] Step S110, obtaining the FFT features corresponding to the original speech data; Figure 8 As shown, a speech enhancement model is constructed, original speech data is collected, and the original speech data is processed by FFT (Fast Fourier Transform) to obtain FFT features.

[0061] Extract FFT features with different window lengths, apply scaling factors to the speech amplitude spectrum, and calculate the corresponding complex spectrum; then calculate the corresponding MSE loss for the amplitude and complex spectrum; by using scaling factors, the difference between the values ​​of the spectrogram can be reduced, so that the model can effectively focus on the details and improve the degree of noise suppression. Among them, the MSE function expression is N refers to the number of features, y i Refers to the real data used in data production. Refers to the prediction result of the model.

[0062] Step S120: extract the FFT features based on the convolutional neural network to obtain convolution feature information.

[0063] Specifically, the convolution module has five layers in total. The number of channels in the first three layers is set to 16. The last two layers are grouped convolutions. The number of groups is equal to the number of input channels, which is 16. The first three layers of convolution are used to extract the FFT features, and then the last two layers of convolution are used to obtain the convolution feature information. Figure 2 As shown, in the first three layers of convolution in the convolution module, the convolution kernel of the first layer is defined as (5, 2), the convolution kernels of the last two layers are (5, 1), and the step size is defined as (2, 1).

[0064] like Figure 3 As shown in the figure, in one of the grouped convolutions of the last two layers of convolution in the convolution module, the input is first upsampled in the channel dimension, the upsampling kernel is defined as (3, 1), and a 1*1 convolution is used to compress the preliminary features to the same number of input channels. Then, after convolution processing with a convolution kernel of (3, 3) and a stride of (1, 1), it is added to the input of the grouped convolution to construct a residual network.

[0065] Step S130: Process the convolution feature information based on a recurrent neural network to obtain processed convolution feature information.

[0066] The recurrent neural network includes an RNN module. The recurrent neural network accepts the output of the convolution module in the convolutional neural network as the input feature of the RNN module. The output of the convolution module is calculated using LSTM in units of channels to obtain the LSTM input features of each channel. This method can effectively retain the structural information of the speech spectrum, and at the same time, the number of parameters required for the recurrent neural network after flattening is smaller. In order to ensure the causal relationship of the model, the gradient calculation of the recurrent neural network is stable, and the influence of the speech input amplitude on the model can also be reduced.

[0067] Among them, Figure 4 As shown in the figure, it should be noted that since the RNN module in the speech enhancement model structure occupies the main parameters, in order to further reduce the number of model parameters, the LSTM unit calculated on each channel uses the same set of parameters. The concatenated LSTM output is processed using the CLN normalization method, that is, the mean and variance are calculated only in the feature and channel dimensions to normalize the concatenated LSTM output, and then the processed convolution feature information is obtained.

[0068] Specifically, the method of step S130 includes the following steps:

[0069] Define the input feature of LSTM on each channel as X, inter is the set of each LSTM input feature concatenated in the channel dimension, concat represents the concatenation operation, and the specific calculation formula satisfies:

[0070]

[0071] The concatenated set of LSTM input features in the channel dimension is normalized after calculating the mean and variance in the feature and channel dimensions to obtain the processed convolution feature information; where:

[0072] Mean

[0073] variance

[0074] The processed convolution feature information Y satisfies the following formula:

[0075]

[0076] Among them, ξ is a very small constant, λ and β are trainable parameters that play a role of scaling and translation; c, f, and X are defined as the number of channels, the number of features, and the LSTM input features, respectively.

[0077] In step S130, the RNN modules constructed in three forms (a structure, b structure, c structure) are compared with the RNN module constructed in this application (d structure). The total parameter amount is shown in Table 1. It can be seen that the RNN module constructed in this application greatly reduces the parameter amount compared with the traditional RNN module.

[0078] a structure, the input is flattened and sent to the LSTM unit. The input and output shapes are the same, that is, the LSTM input and output feature shapes are both (1, 1, 528), and the shape is converted to (1, 16, 33, 1).

[0079] b structure, the input is flattened and sent to the LSTM unit, and the hidden state size is defined as 66, that is, the LSTM input feature shape is (1, 1, 528) and the output feature shape is (1, 1, 66), and then scaled to (1, 1, 528) through the linear layer, and the shape is converted to (1, 16, 33, 1).

[0080] c structure, define an LSTM unit for each channel. Then there are 16 sets of (1, 1, 33) features as input, and the 16 sets of outputs are concatenated to get (1, 16, 33, 1)

[0081] d structure, also grouped by channels, but the LSTM units of all channels share the same set of parameters.

[0082] Specifically, the core of LSTM lies in its unit structure, which includes a forget gate, an input gate, an output gate, and a memory unit, as described below:

[0083] Forget Gate t :

[0084] f t =σ(W f ·[h t-1 , x t ]+b f )

[0085] Among them, f t Determines how much of the current memory state needs to be updated from the previous memory, W f is the weight matrix of the forget gate, b f is the bias, σ is the sigmoid activation function, h t-1 is the hidden state at the previous moment, x t is the input at the current moment.

[0086] Input Gate i t and candidate memory meta-states

[0087] i t =σ(W i·[h t-1 , x t ]+b i )

[0088]

[0089] i t Determines how much new information is written into memory at the current moment,

[0090] is generated at the current moment, indicating the memory that needs to be updated.

[0091] W i and W C are the weight matrices of the input gate and candidate memory states, b i and b c is bias.

[0092] Memory cell state update C t :

[0093]

[0094] C t is the current state of the memory unit, C t-1 is the state of the memory unit at the previous moment.

[0095] Output gate o t And the hidden state update h t :

[0096] o t =σ(W o ·[h t-1 , x t ]+b o )

[0097] h t =o i tanh(C t )

[0098] o t Determines which parts of the memory cell are output, h t is the hidden state at the current moment, which is used to output and pass to the next time step.

[0099] Taking the input feature (1, 16, 33, 1) as an example, if the input feature is flattened, the feature size is (1, 1, 528). From the above formula, we can see that the number of LSTM unit parameters is W f W i W C W o b f bi b C b o Composition, W f The shape is (528*2,528), b f The shape is (1,528), and it can be deduced that the RNN module constructed in this way has 2234496 parameters for the LSTM unit. Similarly, the parameter sizes of the RNN modules constructed with other structures are 192.7k, 143.6k, and 8.9k respectively.

[0100] Table 1

[0101] structure Parameter quantity a 2234.4K b 192.7k C 143.6K d 8.9K

[0102] Step S140, based on different levels of feature fusion (FFA) structure combined with convolutional neural network to obtain the weight coefficients of encoding features and decoding features, multiplying the encoding features and decoding features with the corresponding weight coefficients respectively and then adding them to obtain feature fusion information; wherein, the convolutional neural network includes a convolutional neural network encoder and a convolutional neural network decoder. This method can effectively distinguish the importance of encoding features and decoding features, so that the speech enhancement model can extract more effective features.

[0103] In this embodiment, step S140 may be described as being located before step S130 , and may also be described as being located after step S130 as required.

[0104] Specifically, Figure 5 As shown in the figure, B, C, F, and T represent training batch, channel, feature, and time respectively.

[0105] The method of step S140 comprises the following steps:

[0106] The encoding feature ef and the decoding feature df are concatenated and input into a two-layer convolution with a kernel of 1*1 to obtain the mask features of encoding and decoding;

[0107] The obtained encoded and decoded mask features are mapped to [0, 1] through the Sigmoid function to obtain the weight coefficient corresponding to the encoded feature;

[0108] Multiply the encoding features and decoding features by the corresponding weight coefficients respectively and then add them together to obtain the feature fusion information e n =(Sigmoid(k(ef,df)))*ef+(1-Sigmoid(k(ef,df)))*df.

[0109] Step S150: extracting features from the feature fusion information and the processed convolution feature information based on a convolutional neural network to obtain deconvolution feature information.

[0110] Specifically, the structure of the deconvolution module is symmetrical with that of the convolution module. The deconvolution module has five layers in total. The first two layers are group convolutions, and the last three layers are deconvolution layers that are symmetrically set with the convolution layers of the convolution module. Preliminary feature extraction is performed through the first two layers of group convolution, and then the last three layers of deconvolution are used to obtain deconvolution feature information.

[0111] Among them, Figure 6 As shown in the figure, the input is the output of the previous layer and the output of the convolution layer with symmetrical model structure, and then the features are output by the feature fusion structure at different levels. The concatenated input is also upsampled with the upsampling kernel size of (3,1), and a 1*1 convolution is used to compress the features to half of the input channel. It is then convolved with a convolution with the number of channels set to 16, a kernel of (3,3), and a stride of (1,1), and then added to the input of the previous layer to form a residual network.

[0112] Step S160, based on the deconvolution feature information, multi-frame mask features are obtained, and the output result of the current frame is obtained by combining the multi-frame input features. Using the multi-frame input feature information to obtain the output result of the current frame can effectively reduce noise jitter compared to using only a single-frame mask solution, and effectively improve the quality of the human voice signal in the speech data.

[0113] Specifically, the method of step S160 includes the following steps:

[0114] The deconvolution output feature is represented by E, and the mask feature mask is represented by m. The relationship between the mask feature m and the deconvolution output feature E satisfies:

[0115] m = tanh(E);

[0116] Concatenate the input of the current frame with the input of the previous frame and the input of the next frame to obtain the input feature x of the current frame;

[0117] The input feature x of the current frame is multiplied by the output mask feature m to obtain the output result of the current frame; the input feature x of the current frame is multiplied by the output mask feature m through deep filtering. The calculation expression is as follows: Wherein, x(ct) represents the input feature of the corresponding frame, m(ct) represents the multi-frame mask feature, ct represents the number of observed frames, and in this embodiment, ct=3.

[0118] like Figure 1 As shown, in order to make the technical solution of the present invention clearer, a preferred embodiment is described below.

[0119] Step S110, obtaining FFT features corresponding to the original speech data;

[0120] Step S120: extracting the FFT features based on the convolutional neural network to obtain convolution feature information;

[0121] Step S130: Process the convolution feature information based on a recurrent neural network to obtain processed convolution feature information;

[0122] Step S140, based on the different levels of feature fusion (FFA) structure combined with the convolutional neural network to obtain the weight coefficients of the encoding features and the decoding features, the encoding features and the decoding features are multiplied by the corresponding weight coefficients respectively and then added to obtain feature fusion information;

[0123] Step S150: extracting features from the feature fusion information and the processed convolution feature information based on a convolutional neural network to obtain deconvolution feature information;

[0124] Step S160: derive multi-frame mask features based on the deconvolution feature information, and obtain the output result of the current frame in combination with the multi-frame input features.

[0125] The invention discloses a speech enhancement method based on deep learning. In order to solve the problems of model parameters and performance, the main parts of the specific implementation are as follows: based on convolutional neural networks and recurrent neural networks, channel calculation can effectively retain the structural information of speech signals, and at the same time, a channel feature normalization method can be used to make the model training more stable and reduce the influence of the output speech amplitude. Since the recurrent neural network module parameters account for the main part of the overall model parameters, the parameter sharing method of LSTM units can achieve the purpose of reducing the overall model parameter amount, and the attention mechanism is used for encoding and decoding information to make the model pay more attention to important information parts more effectively. In addition, a deep filtering method is used to link the previous and next speech frame information, so as to further improve the output speech signal quality.

[0126] Figure 7 is a structural block diagram of a deep learning-based speech enhancement system provided by an embodiment of the present invention, such as Figure 7As shown, corresponding to the above-mentioned speech enhancement method based on deep learning, the present invention also provides a speech enhancement system based on deep learning, which includes a module for executing the above-mentioned speech enhancement method based on deep learning. The system can be configured in a terminal such as an electronic device. The speech enhancement system based on deep learning of the present invention is applied. The channel-based calculation based on the convolutional neural network and the recurrent neural network can effectively retain the structural information of the speech signal. At the same time, the channel feature normalization method can make the model training more stable and reduce the impact of the output speech amplitude. The recurrent neural network module is an important component of the model parameter quantity. Since the recurrent neural network module parameters account for the main part of the overall model parameters, the purpose of reducing the overall parameter quantity can be achieved by using the parameter sharing method of the LSTM unit, and the attention mechanism is used for the encoding and decoding information to make the model more effectively focus on the important information part. In addition, the deep filtering method is used to link the previous and next speech frame information, thereby further improving the output speech signal quality.

[0127] Specifically, Figure 7 As shown, the deep learning-based speech enhancement system includes an FFT module 110, a convolution module 120, an RNN module 130, an FFA module 140, a deconvolution module 150 and a deep filtering module 160.

[0128] The FFT module 110 is used to obtain the FFT features corresponding to the original speech data;

[0129] The convolution module 120 is used to extract the FFT features based on the convolutional neural network to obtain convolution feature information;

[0130] The RNN module 130 is used to process the convolution feature information based on a recurrent neural network to obtain the processed convolution feature information;

[0131] The FFA module 140 is used to obtain the weight coefficients of the encoding features and the decoding features based on the different-level feature fusion (FFA) structure combined with the convolutional neural network, and multiply the encoding features and the decoding features by the corresponding weight coefficients respectively and then add them to obtain feature fusion information;

[0132] The deconvolution module 150 is used to extract features from the feature fusion information and the processed convolution feature information based on a convolutional neural network to obtain deconvolution feature information;

[0133] The depth filtering module 160 is used to obtain the mask features of multiple frames based on the deconvolution feature information, and obtain the output result of the current frame in combination with the input features of multiple frames.

[0134] In one embodiment, the deep learning-based speech enhancement system also includes a data acquisition module for acquiring original speech data.

[0135] In one embodiment, the deep learning-based speech enhancement system also includes a construction module for constructing a speech enhancement model.

[0136] In summary, the speech enhancement method and system based on deep learning of the present invention can effectively retain the structural information of the speech signal by calculating based on channels based on convolutional neural networks and recurrent neural networks. At the same time, the channel feature normalization method can make the model training more stable and reduce the impact of the output speech amplitude. Since the recurrent neural network module parameters account for the main part of the overall model parameters, the LSTM unit uses parameter sharing to achieve the purpose of reducing the overall model parameter amount, and the attention mechanism is used for the encoding and decoding information to make the model more effectively focus on the important information part. In addition, the deep filtering method is used to link the previous and next speech frame information, thereby further improving the output voice signal quality.

[0137] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0138] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0139] The steps in the method of the embodiment of the present invention can be adjusted in order, merged and deleted according to actual needs. The units in the device of the embodiment of the present invention can be merged, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present invention can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a terminal, or a network device, etc.) to perform all or part of the steps of the method described in the various embodiments of the present invention.

[0140] The above-described embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A speech enhancement method based on deep learning, characterized in that: The steps include: Get the FFT features corresponding to the original speech data; Extract FFT features based on convolutional neural network to obtain convolution feature information; Processing the convolution feature information based on a recurrent neural network to obtain the processed convolution feature information; Based on the feature fusion structure of different levels combined with the convolutional neural network, the weight coefficients of the encoding features and decoding features are obtained, and the encoding features and decoding features are multiplied by the corresponding weight coefficients respectively and then added to obtain the feature fusion information; Based on the convolutional neural network, feature fusion information and processed convolution feature information are extracted to obtain deconvolution feature information; The multi-frame mask features are obtained based on the deconvolution feature information, and the output result of the current frame is obtained by combining the multi-frame input features.

2. The deep learning-based speech enhancement method according to claim 1, characterized in that: Before the step of obtaining the FFT features corresponding to the original voice data, the step further includes: Build a speech enhancement model.

3. The deep learning-based speech enhancement method according to claim 1, characterized in that: The method for extracting the FFT features based on the convolutional neural network to obtain the convolution feature information includes the following steps: The convolution module has a total of five layers. The number of channels in the first three layers is set to 16, and the last two layers are grouped convolutions. The number of groups is equal to the number of input channels, which is 16. The first three layers of convolution are used to perform preliminary feature extraction on the FFT features, and then the last two layers of convolution are used to obtain the convolution feature information.

4. The deep learning-based speech enhancement method according to claim 3, characterized in that: The step processes the convolution feature information based on a recurrent neural network to obtain the processed convolution feature information, comprising the following steps: Define the input feature of LSTM on each channel as X, inter is the set of each LSTM input feature concatenated in the channel dimension, concat represents the concatenation operation, and the specific calculation formula satisfies: The concatenated set of LSTM input features in the channel dimension is normalized after calculating the mean and variance in the feature and channel dimensions to obtain the processed convolution feature information; where: Mean variance The processed convolution feature information Y satisfies the following formula: Among them, ξ is a constant, λ and β are trainable parameters, and c, f, and X are defined as the number of channels, the number of features, and the LSTM input features, respectively.

5. The deep learning-based speech enhancement method according to claim 1, characterized in that: The steps are based on different levels of feature fusion structures combined with convolutional neural networks to obtain weight coefficients of encoding features and decoding features, and the encoding features and decoding features are multiplied by the corresponding weight coefficients and then added to obtain feature fusion information. The specific operations include: The encoding feature ef and the decoding feature df are concatenated and input into a two-layer convolution with a kernel of 1*1 to obtain the mask features of encoding and decoding; The obtained encoded and decoded mask features are mapped to [0, 1] through the Sigmoid function to obtain the weight coefficient corresponding to the encoded feature; Multiply the encoding features and decoding features by the corresponding weight coefficients respectively and then add them together to obtain the feature fusion information e n =(Sigmoid(k(ef,df)))*ef+(1-Sigmoid(k(ef,df)))*df.

6. The method for speech enhancement based on deep learning according to claim 1, characterized in that: The step extracts features from feature fusion information and processed convolution feature information based on a convolutional neural network to obtain deconvolution feature information. The specific operations include: The structure of the deconvolution module is symmetrical with that of the convolution module. The deconvolution module has five layers in total. The first two layers are group convolutions, and the last three layers are deconvolution layers that are symmetrically set with the convolution layers of the convolution module. Preliminary feature extraction is performed through the first two layers of group convolution, and then the last three layers of deconvolution are used to obtain deconvolution feature information.

7. The method for speech enhancement based on deep learning according to claim 1, characterized in that: The step is to obtain the multi-frame mask features based on the deconvolution feature information, and to obtain the output result of the current frame by combining the multi-frame input features. The specific operations include: The deconvolution output feature is represented by E, and the mask feature mask is represented by m. The relationship between the mask feature m and the deconvolution output feature E satisfies: m = tanh(E); Concatenate the input of the current frame with the input of the previous frame and the input of the next frame to obtain the input feature x of the current frame; The input feature x of the current frame is dot-multiplied with the output mask feature m to obtain the output result of the current frame.

8. The deep learning-based speech enhancement method according to claim 7, characterized in that: The step performs a dot product sum of the input feature x of the current frame and the output mask feature m to obtain the output result of the current frame. The specific operations include: The input feature x of the current frame is summed by the dot multiplication of the output mask feature m through deep filtering. The calculation expression is as follows: Among them, x(ct) represents the input features of the corresponding frame, m(ct) represents the multi-frame mask features, and ct represents the number of observed frames.

9. A speech enhancement system based on deep learning, characterized in that: include FFT module, used to obtain FFT features corresponding to original speech data; The convolution module is used to extract the FFT features based on the convolutional neural network and obtain the convolution feature information; The RNN module is used to process the convolution feature information based on the recurrent neural network and obtain the processed convolution feature information; The FFA module is used to obtain the weight coefficients of the encoding features and decoding features based on the feature fusion structures at different levels combined with the convolutional neural network, and multiply the encoding features and decoding features with the corresponding weight coefficients respectively and then add them to obtain the feature fusion information; A deconvolution module is used to extract features from feature fusion information and processed convolution feature information based on a convolutional neural network to obtain deconvolution feature information; The deep filtering module is used to obtain multi-frame mask features based on the deconvolution feature information and obtain the output result of the current frame by combining the multi-frame input features.

10. The deep learning-based speech enhancement system according to claim 9, characterized in that: Also included are building blocks for constructing speech enhancement models.

Images