Speech enhancement method and system based on deep learning

Abstract

The application discloses a speech enhancement method and system based on deep learning, which comprises the following steps: acquiring FFT features corresponding to original speech data; performing feature extraction on the FFT features based on a convolutional neural network to acquire convolutional feature information; processing the convolutional feature information based on a recurrent neural network to acquire processed convolutional feature information; combining the convolutional neural network based on a different-level feature fusion structure to acquire weight coefficients of encoding features and decoding features and to acquire feature fusion information; performing feature extraction on the feature fusion information and the processed convolutional feature information based on the convolutional neural network to acquire deconvolutional feature information; and obtaining multi-frame mask features based on the deconvolutional feature information and combining the multi-frame input features to obtain an output result of a current frame. The application can effectively retain the structural information of a speech signal by performing calculation in a channel based on the convolutional neural network and the recurrent neural network, and can achieve the purpose of reducing the overall parameter quantity of a model by a parameter sharing mode.

Description

Technical Field

[0001] This invention relates to the field of audio information processing technology, and in particular to a speech enhancement method and system based on deep learning. Background Technology

[0002] Traditional single-channel speech enhancement methods typically require making assumptions about the interaction between noise and clean speech signals. These assumptions are often based on specific mathematical models or statistical properties, which limits the performance and applicability of the enhancement techniques, resulting in unsatisfactory enhanced speech quality. In contrast, deep learning-based methods do not require explicit assumptions about the relationship between speech and noise signals. These methods utilize large amounts of training data to directly learn the mapping relationship between clean speech and noise. The advantage of this approach lies in its ability to automatically extract features from data and learn complex nonlinear relationships, thereby overcoming some inherent problems of traditional speech enhancement algorithms and achieving better denoising results and stronger generalization capabilities.

[0003] However, this also brings new challenges. Deep learning algorithms often require a large number of parameters and computational overhead. The computational overhead problem in embedded devices has been somewhat solved with the development of chip performance. The size of model parameters often restricts the widespread application of deep learning models in devices, and the number of parameters is often closely related to the performance of the model.

[0004] Therefore, it is urgent to propose a speech enhancement method to solve the problem. Summary of the Invention

[0005] Therefore, it is necessary to address the shortcomings of existing technologies by providing a deep learning-based speech enhancement method and system that reduces the overall number of model parameters through parameter sharing.

[0006] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0007] Firstly, a deep learning-based speech enhancement method is provided, which includes the following steps:

[0008] Obtain the FFT features corresponding to the raw speech data;

[0009] Feature extraction is performed on FFT features based on convolutional neural networks to obtain convolutional feature information;

[0010] The convolutional feature information is processed based on a recurrent neural network to obtain the processed convolutional feature information;

[0011] Based on the feature fusion structure at different levels, combined with the convolutional neural network, the weight coefficients of the encoded features and decoded features are obtained. The encoded features and decoded features are multiplied by their corresponding weight coefficients and then added together to obtain the feature fusion information.

[0012] Based on convolutional neural networks, feature extraction is performed on feature fusion information and processed convolutional feature information to obtain deconvolutional feature information;

[0013] The mask features of multiple frames are obtained based on the deconvolution feature information, and the output result of the current frame is obtained by combining the input features of multiple frames.

[0014] In one embodiment, before obtaining the FFT features corresponding to the original speech data, the method further includes:

[0015] Build a speech enhancement model.

[0016] In one embodiment, the method for extracting features from FFT features using a convolutional neural network to obtain convolutional feature information includes the following steps:

[0017] The convolutional module has a total of five layers. The first three convolutional layers have 16 channels, and the last two convolutional layers are grouped convolutions. The number of groups is equal to the number of input channels, and the number of groups is 16. The first three convolutional layers are used to perform preliminary feature extraction on the FFT features, and then the last two convolutional layers are used to obtain convolutional feature information.

[0018] In one embodiment, the method for processing convolutional feature information based on a recurrent neural network to obtain the processed convolutional feature information includes the following steps:

[0019] Let X be the input feature of the LSTM in each channel, inter be the set of concatenated LSTM input features along the channel dimension, and concat represent the concatenation operation. Then the specific calculation formula satisfies:

[0020]

[0021] The concatenated set of LSTM input features along the channel dimension is normalized after calculating the mean and variance along both the feature and channel dimensions to obtain the processed convolutional feature information; where...

[0022] mean

[0023] variance

[0024] The processed convolutional feature information Y satisfies the following formula:

[0025]

[0026] Where ξ is a constant, λ and β are trainable parameters, and c, f, and X are defined as the number of channels, the number of features, and the LSTM input features, respectively.

[0027] In one embodiment, the step of obtaining weight coefficients for encoded and decoded features based on different-level feature fusion structures combined with convolutional neural networks, and then multiplying the encoded and decoded features by their corresponding weight coefficients and adding them together to obtain feature fusion information, specifically includes:

[0028] The encoded feature ef and the decoded feature df are concatenated and then input into a convolution with two 1*1 kernels to obtain the mask features for encoding and decoding.

[0029] The obtained encoding and decoding mask features are mapped to [0, 1] using the Sigmoid function to obtain the weight coefficients corresponding to the encoding features;

[0030] The feature fusion information e is obtained by multiplying the encoded features and decoded features by their respective weight coefficients and then summing them. n =(Sigmoid(k(ef,df)))*ef+(1-Sigmoid(k(ef,df)))*df.

[0031] In one embodiment, the step of extracting features from feature fusion information and processed convolutional feature information based on a convolutional neural network to obtain deconvolutional feature information specifically includes:

[0032] The deconvolution module is symmetrical to the convolution module in structure. The deconvolution module has a total of five layers. The first two layers are grouped convolutions, and the last three layers are deconvolution layers set symmetrically with the convolution layers of the convolution module. The first two layers of grouped convolutions are used for preliminary feature extraction, and then the last three layers of deconvolutions are used to obtain deconvolution feature information.

[0033] In one embodiment, the method of obtaining multi-frame mask features based on deconvolution feature information and combining multi-frame input features to obtain the output result of the current frame specifically includes the following operations:

[0034] Let E represent the deconvolution output feature and m represent the mask feature. The relationship between the mask feature m and the deconvolution output feature E satisfies:

[0035] m = tanh(E);

[0036] The input of the current frame is concatenated with the input of the previous frame and the input of the next frame to obtain the input feature x of the current frame;

[0037] The output of the current frame is obtained by multiplying the input feature x of the current frame with the output mask feature m.

[0038] In one embodiment, the method of summing the dot product of the input feature x of the current frame and the output mask feature m to obtain the output result of the current frame specifically includes the following operations:

[0039] The input feature x of the current frame is multiplied by the output mask feature m using depth filtering, and the summation is expressed as follows: Where x(ct) represents the input feature of the corresponding frame, m(ct) represents the multi-frame mask feature, and ct represents the number of observed frames.

[0040] Secondly, a deep learning-based speech enhancement system is provided, which includes...

[0041] The FFT module is used to obtain the FFT features corresponding to the original speech data;

[0042] The convolution module is used to extract features from FFT features based on convolutional neural networks and obtain convolutional feature information;

[0043] The RNN module is used to process convolutional feature information based on recurrent neural networks and obtain the processed convolutional feature information.

[0044] The FFA module is used to obtain the weight coefficients of encoded and decoded features based on different levels of feature fusion (FFA) structure combined with convolutional neural networks. The encoded features and decoded features are multiplied by their corresponding weight coefficients and then added together to obtain feature fusion information.

[0045] The deconvolution module is used to extract features from the feature fusion information and the processed convolutional feature information based on the convolutional neural network, and to obtain deconvolutional feature information.

[0046] The depth filtering module is used to obtain the mask features of multiple frames based on the deconvolution feature information, and to obtain the output result of the current frame by combining the input features of multiple frames.

[0047] In one embodiment, a building module is also included for building a speech enhancement model.

[0048] In summary, the speech enhancement method and system based on deep learning of this invention can effectively preserve the structural information of speech signals by performing channel-based calculations using convolutional neural networks and recurrent neural networks. Simultaneously, the channel feature normalization method makes the model training more stable and reduces the impact of output speech amplitude. Since the parameters of the recurrent neural network module constitute the main part of the overall model parameters, the parameter sharing method of the LSTM unit can reduce the overall number of model parameters. Furthermore, the attention mechanism applied to the encoding and decoding information allows the model to more effectively focus on important information parts. In addition, the use of deep filtering to link the information of consecutive speech frames further improves the quality of the output speech signal. Attached Figure Description

[0049] Figure 1 This is a flowchart illustrating a deep learning-based speech enhancement method provided in an embodiment of the present invention.

[0050] Figure 2 This is a flowchart illustrating step S120 provided in an embodiment of the present invention;

[0051] Figure 3 This is a flowchart illustrating the grouped convolution operation in step S120 provided in an embodiment of the present invention;

[0052] Figure 4 This is a flowchart illustrating step S130 provided in an embodiment of the present invention;

[0053] Figure 5 This is a flowchart illustrating step S140 provided in an embodiment of the present invention;

[0054] Figure 6 This is a flowchart illustrating step S150 provided in an embodiment of the present invention;

[0055] Figure 7 This is a structural block diagram of a deep learning-based speech enhancement system provided in an embodiment of the present invention;

[0056] Figure 8 This is a structural principle diagram of a speech enhancement model provided in an embodiment of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] It should be noted that the terms "first," "second," "third," "fourth," etc., in the specification and claims of this invention are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "having," and any variations thereof, in the embodiments of this invention are intended to cover non-exclusive inclusion. Exemplarily, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0059] like Figure 1This is a flowchart illustrating a deep learning-based speech enhancement method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, this deep learning-based speech enhancement method includes steps S110-S160, as detailed below:

[0060] Step S110: Obtain the FFT features corresponding to the original speech data; such as Figure 8 As shown, a speech enhancement model is constructed, raw speech data is collected, and FFT (Fast Fourier Transform) is used to process the raw speech data to obtain FFT features.

[0061] FFT features with different window lengths are extracted, a scaling factor is applied to the speech amplitude spectrum, and the corresponding complex spectrum is calculated. Then, the MSE loss is calculated for the amplitude and complex spectra. By using the scaling factor, the differences between spectrogram values ​​can be reduced, allowing the model to effectively focus on details and improve noise suppression. The MSE function expression is as follows: N represents the characteristic number, y i Refers to the actual data used in data creation. Refers to the prediction results of the model.

[0062] Step S120: Extract features from FFT features based on convolutional neural networks to obtain convolutional feature information.

[0063] Specifically, the convolutional module consists of five layers. The first three convolutional layers have 16 channels each, while the last two layers are grouped convolutions with the same number of groups as the input channels (16 groups in total). The first three convolutional layers perform preliminary feature extraction from the FFT features, and the last two convolutional layers then provide the final convolutional feature information. For example... Figure 2 As shown, in the first three convolutional layers of the convolutional module, the first convolutional kernel is defined as (5, 2), the last two convolutional kernels are (5, 1), and the stride is defined as (2, 1).

[0064] like Figure 3 As shown, in one of the group convolutions of the last two convolutional layers of the convolutional module, the input is first upsampled in terms of channel dimension, with the upsampling kernel defined as (3, 1), and the initial features are compressed to be comparable to the number of input channels through a 1*1 convolution. Then, after being processed by a convolution with a kernel of (3, 3) and a stride of (1, 1), it is added to the input of the group convolution to construct a residual network.

[0065] Step S130: Process the convolutional feature information based on the recurrent neural network to obtain the processed convolutional feature information.

[0066] Recurrent Neural Networks (RNNs) include RNN modules. The RNN accepts the output of the convolutional modules in a convolutional neural network as the input features of the RNN module. The output of the convolutional modules is processed by LSTM on a channel-by-channel basis to obtain the LSTM input features of each channel. This method can effectively preserve the structural information of the speech spectrum, while requiring fewer parameters than a flattened RNN. Furthermore, it ensures the stability of gradient calculation in the RNN to maintain the causal relationship of the model, and also reduces the impact of speech input amplitude on the model.

[0067] Among them, such as Figure 4 As shown, it's important to note that since the RNN module occupies the majority of parameters in this speech enhancement model structure, the LSTM units computed on each channel use the same set of parameters to further reduce the number of model parameters. Furthermore, CLN normalization is used to process the concatenated LSTM output; that is, the mean and variance are calculated only in the feature and channel dimensions to normalize the concatenated LSTM output, thereby obtaining the processed convolutional feature information.

[0068] Specifically, the method of step S130 includes the following steps:

[0069] Let X be the input feature of the LSTM in each channel, inter be the set of concatenated LSTM input features along the channel dimension, and concat represent the concatenation operation. Then the specific calculation formula satisfies:

[0070]

[0071] The concatenated set of LSTM input features along the channel dimension is normalized after calculating the mean and variance along both the feature and channel dimensions to obtain the processed convolutional feature information; where...

[0072] mean

[0073] variance

[0074] The processed convolutional feature information Y satisfies the following formula:

[0075]

[0076] Here, ξ is a very small constant, λ and β are trainable parameters that serve as a scaling and translation function; c, f, and X are defined as the number of channels, the number of features, and the LSTM input features, respectively.

[0077] In step S130, the RNN modules constructed in three forms (structure a, structure b, and structure c) are compared with the RNN module constructed in this application (structure d). The total number of parameters is shown in Table 1. It can be seen that the RNN module constructed in this application significantly reduces the number of parameters compared to the traditional RNN module.

[0078] Structure a: After flattening the input, it is fed into the LSTM unit. The input and output shapes are the same, that is, the LSTM input and output feature shapes are both (1, 1, 528), and the shape is transformed into (1, 16, 33, 1).

[0079] The b structure flattens the input and feeds it into the LSTM unit. The hidden state size is defined as 66, that is, the LSTM input feature shape is (1, 1, 528) and the output feature shape is (1, 1, 66). Then, it is scaled to (1, 1, 528) through a linear layer, and the shape is transformed into (1, 16, 33, 1).

[0080] The structure is c, and an LSTM unit is defined for each channel. There are 16 sets of (1, 1, 33) features as input, and the 16 sets of outputs are concatenated to obtain (1, 16, 33, 1).

[0081] The d structure is also grouped by channels, but all channels of the LSTM unit share the same set of parameters.

[0082] Specifically, the core of LSTM lies in its cell structure, which includes forget gates, input gates, output gates, and memory cells, as described below:

[0083] Forgotten Gate f t :

[0084] f t =σ(W f ·[h t-1 x t ]+b f )

[0085] Among them, f t W determines how much of the current memory state needs to be retrieved from the previous moment. f This is the weight matrix of the forget gate, b f It is the bias, σ is the sigmoid activation function, and h t-1 It is the hidden state from the previous moment, x t It is the input at the current moment.

[0086] Input gate i t and candidate memory states

[0087] i t =σ(W i·[h t-1 x t ]+b i )

[0088]

[0089] i t This determines how much of the new information at the current moment is written into memory.

[0090] It is generated at the current moment and represents the memory that needs to be updated.

[0091] W i and W C These are the weight matrices for the input gate and the candidate memory state, respectively. i and b c It is a bias.

[0092] Memory cell state update C t :

[0093]

[0094] C t It is the current state of the memory unit, C. t-1 It is the state of the memory unit at the previous moment.

[0095] Output gate o t And hidden state update h t :

[0096] o t =σ(W o ·[h t-1 x t ]+b o )

[0097] h t =o i ·tanh(C t )

[0098] o t It determines which parts of the memory unit are output, h t It is the hidden state at the current moment, used for output and propagation to the next time step.

[0099] Taking the input feature (1, 16, 33, 1) as an example, if the input feature is flattened, the feature size becomes (1, 1, 528). From the above formula, it can be seen that the number of LSTM unit parameters is W... f W i W C W o b f bi b C b o Composition, W f The shape is (528*2,528), b f With a shape of (1,528), the number of LSTM unit parameters in the RNN module constructed in this way is 2,234,496. Similarly, the number of parameters in the RNN modules constructed with other structures are 192.7k, 143.6k, and 8.9k, respectively.

[0100] Table 1

[0101] a 2234.4K b 192.7k C 143.6K d 8.9K

[0102] Step S140: Based on the feature fusion (FFA) structure at different levels, the weight coefficients of the encoded features and decoded features are obtained by combining convolutional neural networks. The encoded features and decoded features are multiplied by their corresponding weight coefficients and then added together to obtain the feature fusion information. The convolutional neural network includes a convolutional neural network encoder and a convolutional neural network decoder. This method can effectively distinguish the importance of the encoded features and decoded features, so that the speech enhancement model can extract more effective features.

[0103] In this embodiment, step S140 can be described as occurring before step S130, or it can be described as occurring after step S130 as needed.

[0104] Specifically, such as Figure 5 As shown in the figure, B, C, F, and T represent the training batch, channel, feature, and time, respectively.

[0105] The method of step S140 includes the following steps:

[0106] The encoded feature ef and the decoded feature df are concatenated and then input into a convolution with two 1*1 kernels to obtain the mask features for encoding and decoding.

[0107] The obtained encoding and decoding mask features are mapped to [0, 1] using the Sigmoid function to obtain the weight coefficients corresponding to the encoding features;

[0108] The feature fusion information e is obtained by multiplying the encoded features and decoded features by their respective weight coefficients and then summing them. n =(Sigmoid(k(ef,df)))*ef+(1-Sigmoid(k(ef,df)))*df.

[0109] Step S150: Based on the convolutional neural network, feature extraction is performed on the feature fusion information and the processed convolutional feature information to obtain deconvolutional feature information.

[0110] Specifically, the deconvolution module is symmetrical to the convolution module in structure. The deconvolution module has a total of five layers. The first two layers are grouped convolutions, and the last three layers are deconvolution layers that are symmetrically set with the convolution layers of the convolution module. The first two layers of grouped convolutions are used for preliminary feature extraction, and then the last three layers of deconvolutions are used to obtain deconvolution feature information.

[0111] Among them, such as Figure 6 As shown, the input is the output of the previous layer and the output of the convolutional layer with a symmetrical model structure, which are concatenated and then processed by the features output by the feature fusion structure at different levels. The concatenated input is then upsampled with a kernel size of (3,1) and compressed to half the input channel size using a 1*1 convolution. After convolution with a convolutional layer with 16 channels, a kernel size of (3,3), and a stride of (1,1), the input is added to the input of the previous layer to form a residual network.

[0112] Step S160: Calculate the multi-frame mask features based on the deconvolution feature information, and combine the multi-frame input features to obtain the output result of the current frame. Using the multi-frame input feature information to obtain the output result of the current frame can effectively reduce noise jitter and effectively improve the quality of human voice signals in speech data compared to using only a single-frame mask.

[0113] Specifically, the method of step S160 includes the following steps:

[0114] Let E represent the deconvolution output feature and m represent the mask feature. The relationship between the mask feature m and the deconvolution output feature E satisfies:

[0115] m = tanh(E);

[0116] The input of the current frame is concatenated with the input of the previous frame and the input of the next frame to obtain the input feature x of the current frame;

[0117] The output of the current frame is obtained by multiplying the input feature x of the current frame with the output mask feature m and summing the results. The summation of the input feature x of the current frame with the output mask feature m using depth filtering is expressed as follows: Where x(ct) represents the input feature of the corresponding frame, m(ct) represents the multi-frame mask feature, and ct represents the number of observed frames. In this embodiment, ct = 3.

[0118] like Figure 1 As shown, in order to make the technical solution of the present invention clearer, preferred embodiments will be described below.

[0119] Step S110: Obtain the FFT features corresponding to the original speech data;

[0120] Step S120: Extract features from FFT features based on convolutional neural networks to obtain convolutional feature information;

[0121] Step S130: Process the convolutional feature information based on the recurrent neural network to obtain the processed convolutional feature information;

[0122] Step S140: Based on the feature fusion (FFA) structure at different levels, combine with a convolutional neural network to obtain the weight coefficients of the encoded features and decoded features. Multiply the encoded features and decoded features by their respective weight coefficients and then add them together to obtain the feature fusion information.

[0123] Step S150: Based on the convolutional neural network, perform feature extraction on the feature fusion information and the processed convolutional feature information to obtain deconvolutional feature information;

[0124] Step S160: Calculate the multi-frame mask features based on the deconvolution feature information, and combine the multi-frame input features to obtain the output result of the current frame.

[0125] This invention discloses a speech enhancement method based on deep learning. To address model parameter and performance issues, the main implementation components are as follows: Channel-based computation using convolutional neural networks and recurrent neural networks effectively preserves the structural information of the speech signal. Channel feature normalization further stabilizes model training and reduces the impact of output speech amplitude. Since recurrent neural network module parameters constitute a major portion of the overall model parameters, parameter sharing through LSTM units reduces the overall model parameter count. An attention mechanism is applied to the encoding and decoding information, enabling the model to focus more effectively on important information. Furthermore, deep filtering is used to link preceding and following speech frames, further improving the quality of the output speech signal.

[0126] Figure 7 This is a structural block diagram of a deep learning-based speech enhancement system provided in an embodiment of the present invention, such as... Figure 7As shown, corresponding to the aforementioned deep learning-based speech enhancement method, this invention also provides a deep learning-based speech enhancement system. This system includes a module for executing the deep learning-based speech enhancement method. The system can be configured in terminals such as electronic devices. Applying this invention's deep learning-based speech enhancement system, which calculates by channel using convolutional neural networks and recurrent neural networks, effectively preserves the structural information of the speech signal. Simultaneously, channel feature normalization makes the model training more stable and reduces the impact of output speech amplitude. The recurrent neural network module is a crucial component of the model's parameter count. Since the recurrent neural network module parameters constitute a major portion of the overall model parameters, parameter sharing through LSTM units can reduce the overall parameter count. Furthermore, an attention mechanism is used on the encoding and decoding information, allowing the model to more effectively focus on important information. Additionally, deep filtering is used to link preceding and following speech frames, further improving the quality of the output speech signal.

[0127] Specifically, such as Figure 7 As shown, the deep learning-based speech enhancement system includes an FFT module 110, a convolution module 120, an RNN module 130, an FFA module 140, a deconvolution module 150, and a depth filtering module 160.

[0128] FFT module 110 is used to obtain the FFT features corresponding to the original speech data;

[0129] Convolutional module 120 is used to extract features from FFT features based on convolutional neural networks and obtain convolutional feature information;

[0130] RNN module 130 is used to process convolutional feature information based on recurrent neural networks and obtain the processed convolutional feature information.

[0131] FFA module 140 is used to obtain the weight coefficients of encoded features and decoded features based on different levels of feature fusion (FFA) structure combined with convolutional neural network, and to obtain feature fusion information by multiplying the encoded features and decoded features by their corresponding weight coefficients and then adding them together.

[0132] The deconvolution module 150 is used to extract features from the feature fusion information and the processed convolutional feature information based on the convolutional neural network, and to obtain deconvolutional feature information.

[0133] The depth filtering module 160 is used to obtain the mask features of multiple frames based on the deconvolution feature information, and to obtain the output result of the current frame by combining the input features of multiple frames.

[0134] In one embodiment, the deep learning-based speech enhancement system further includes a data acquisition module for acquiring raw speech data.

[0135] In one embodiment, the deep learning-based speech enhancement system further includes a building module for building a speech enhancement model.

[0136] In summary, the speech enhancement method and system based on deep learning of this invention can effectively preserve the structural information of speech signals by performing channel-based calculations using convolutional neural networks and recurrent neural networks. Simultaneously, the channel feature normalization method makes the model training more stable and reduces the impact of output speech amplitude. Since the parameters of the recurrent neural network module constitute the main part of the overall model parameters, the parameter sharing method of the LSTM unit can reduce the overall model parameter count. Furthermore, the attention mechanism applied to the encoding and decoding information allows the model to more effectively focus on important information parts. In addition, the use of deep filtering to link the information of consecutive speech frames further improves the quality of the output speech signal.

[0137] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0138] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For example, the division of each unit is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0139] The steps in the method of this invention can be adjusted, merged, or deleted according to actual needs. The units in the device of this invention can be merged, divided, or deleted according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which can be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of this invention.

[0140] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A speech enhancement method based on deep learning, characterized in that, Includes the following steps: Obtain the FFT features corresponding to the raw speech data; Feature extraction is performed on FFT features based on convolutional neural networks to obtain convolutional feature information; The convolutional feature information is processed based on a recurrent neural network to obtain the processed convolutional feature information; Based on the feature fusion structure at different levels, combined with the convolutional neural network, the weight coefficients of the encoded features and decoded features are obtained. The encoded features and decoded features are multiplied by their corresponding weight coefficients and then added together to obtain the feature fusion information. Based on convolutional neural networks, feature extraction is performed on feature fusion information and processed convolutional feature information to obtain deconvolutional feature information; Based on the deconvolution feature information, multi-frame mask features are obtained, and combined with the multi-frame input features, the output result of the current frame is obtained. The method for processing convolutional feature information based on a recurrent neural network to obtain the processed convolutional feature information includes the following steps: Let X be the input feature of the LSTM in each channel, inter be the set of concatenated LSTM input features along the channel dimension, and concat represent the concatenation operation. Then the specific calculation formula satisfies: ； The concatenated set of LSTM input features along the channel dimension is normalized after calculating the mean and variance along both the feature and channel dimensions to obtain the processed convolutional feature information; where... mean , variance , The processed convolutional feature information Y satisfies the following formula: ， in, It is a constant. and These are trainable parameters, c. f X is defined as the number of channels, the number of features, and the LSTM input features, respectively. All LSTM units in all channels share the same set of parameters.

2. The speech enhancement method based on deep learning according to claim 1, characterized in that, Before obtaining the FFT features corresponding to the original speech data, the following steps are also included: Build a speech enhancement model.

3. The speech enhancement method based on deep learning according to claim 1, characterized in that, The method for extracting features from FFT features using a convolutional neural network to obtain convolutional feature information includes the following steps: A convolutional neural network includes a convolutional module with a total of five layers. The first three convolutional layers have 16 channels, and the last two convolutional layers are grouped convolutions. The number of groups is equal to the number of input channels, and the number of groups is 16. The first three convolutional layers are used to perform preliminary feature extraction on FFT features, and the last two convolutional layers are used to obtain convolutional feature information.

4. The speech enhancement method based on deep learning according to claim 1, characterized in that, The steps described above are based on different levels of feature fusion structures combined with convolutional neural networks to obtain weight coefficients for encoded and decoded features. The method involves multiplying the encoded and decoded features by their respective weight coefficients and then summing the results to obtain feature fusion information. Specific operations include: The encoded feature ef and the decoded feature df are concatenated and then input into a two-layer convolution with a kernel of 1*1 to obtain the mask features for encoding and decoding. The obtained encoding and decoding mask features are mapped to [0, 1] using the Sigmoid function to obtain the weight coefficients corresponding to the encoding features; The feature fusion information e is obtained by multiplying the encoded features and decoded features by their respective weight coefficients and then summing them. n =(Sigmoid( k (ef,df)))*ef+(1- Sigmoid ( k (ef,df)))*df.

5. The deep learning-based speech enhancement method according to claim 1, characterized in that, The steps described above involve extracting features from the feature fusion information and the processed convolutional feature information using a convolutional neural network to obtain deconvolutional feature information. The specific operations include: Convolutional neural networks include deconvolution modules, which are symmetrical to the convolution modules. The deconvolution module has a total of five layers. The first two layers are grouped convolutions, and the last three layers are deconvolution layers that are symmetrical to the convolutional layers of the convolution modules. The first two layers of grouped convolutions are used for preliminary feature extraction, and the last three layers of deconvolutions are used to obtain deconvolution feature information.

6. The speech enhancement method based on deep learning according to claim 1, characterized in that, The method described above, which calculates multi-frame mask features based on deconvolution feature information and combines them with multi-frame input features to obtain the output result of the current frame, specifically includes the following steps: Setting deconvolution output features This indicates that the mask feature is used... To represent, mask features With deconvolution output features The relationship satisfies: ； The input of the current frame is concatenated with the inputs of the previous and next frames to obtain the input features of the current frame. x ; Input features of the current frame x The output of the current frame is obtained by multiplying the output mask feature m by the summation.

7. The deep learning-based speech enhancement method according to claim 6, characterized in that, The step involves taking the input features of the current frame. x The method for obtaining the output result of the current frame by summing the dot product of the output mask features m and the result of the summation includes the following specific operations: Input features of the current frame x The summation of the dot product of the output mask feature m and the summation is expressed as follows: ,in, x ( ct ) represents the input features of the corresponding frame. m ( ct ) represents multi-frame mask features, Indicates the number of observation frames.

8. A speech enhancement system based on deep learning, characterized in that, include The FFT module is used to obtain the FFT features corresponding to the original speech data; The convolution module is used to extract features from FFT features based on convolutional neural networks and obtain convolutional feature information. The RNN module is used to process convolutional feature information based on recurrent neural networks and obtain the processed convolutional feature information. The FFA module is used to obtain the weight coefficients of encoded features and decoded features based on different levels of feature fusion structures combined with convolutional neural networks. The encoded features and decoded features are multiplied by their corresponding weight coefficients and then added together to obtain feature fusion information. The deconvolution module is used to extract features from the feature fusion information and the processed convolutional feature information based on the convolutional neural network, and to obtain deconvolutional feature information. The depth filtering module is used to obtain the mask features of multiple frames based on the deconvolution feature information, and combine the input features of multiple frames to obtain the output result of the current frame. The RNN module is used to process convolutional feature information based on a recurrent neural network to obtain the processed convolutional feature information, and the method includes the following steps: Let X be the input feature of the LSTM in each channel, inter be the set of concatenated LSTM input features along the channel dimension, and concat represent the concatenation operation. Then the specific calculation formula satisfies: ； The concatenated set of LSTM input features along the channel dimension is normalized after calculating the mean and variance along both the feature and channel dimensions to obtain the processed convolutional feature information; where... mean , variance , The processed convolutional feature information Y satisfies the following formula: ， in, It is a constant. and These are trainable parameters, c. f X is defined as the number of channels, the number of features, and the LSTM input features, respectively. All LSTM units in all channels share the same set of parameters.

9. The deep learning-based speech enhancement system according to claim 8, characterized in that, It also includes building blocks for constructing speech enhancement models.

Images