Attention-based complex convolutional neural network speech enhancement method and system

Abstract

The application provides a complex convolutional neural network speech enhancement method and system based on attention, and has the characteristics that: the speech with background noise is denoised to realize the enhancement of the speech, which comprises capturing the speech distribution in the time-frequency domain by using a channel time-frequency attention mechanism; time modeling is performed by using a gated time convolution network (GTCN), so as to reduce the parameters of the network and improve the training speed of the network; and sub-pixel convolution is used to realize the reconstruction of pure speech, so as to avoid the chessboard artifacts caused by deconvolution. The application effectively improves the network denoising effect, reduces the noise influence in the speech signal, and thus improves the intelligibility and quality of the speech.

Description

TECHNICAL FIELD

[0001] The application relates to the field of speech noise reduction, and particularly relates to a complex convolutional neural network speech enhancement method and system based on attention. BACKGROUND

[0002] In daily life, speech signals are interfered by background noise to cause distortion. The distortion reduces the intelligibility and quality of the speech, affects the understanding of the speech content by people, causes a poor user experience, and brings inconvenience to communication in daily life. To solve the above noise interference problem, a speech enhancement technology is proposed. The speech enhancement technology is a technology for removing or weakening background noise in a speech signal containing noise to recover a pure speech signal, so as to reduce the interference of noise on speech as much as possible and improve the communication quality of speech.

[0003] In recent years, artificial intelligence technology develops rapidly, and deep learning has become a hotspot in many research fields and has achieved many research results. Since the performance of traditional speech enhancement methods such as Wiener filtering, spectral subtraction, and methods based on statistical models is limited, deep learning technology is introduced into the field of speech enhancement and has achieved remarkable results. Speech enhancement is expressed as supervised learning, and the selection of the training target is very important. The early speech enhancement method is mainly based on time-frequency domain analysis method, and the training target is mainly divided into two categories. One is a mask-based training target used to define the time-frequency relationship between clean speech and noise speech. The other is a mapping-based training target representing the spectral features of clean speech. However, these training targets are mostly operated on the amplitude spectrum of noise speech and ignore the phase, that is, the real part is focused and the imaginary part is ignored, resulting in problems such as poor speech signal noise reduction effect. In addition, long-time dependence modeling of speech in the speech enhancement method has been a problem for domestic and foreign researches. SUMMARY

[0004] In view of the above problems of the prior art, the application provides a complex convolutional neural network speech enhancement method and system based on attention.

[0005] In order to solve the above problems in the prior art, the application provides a complex convolutional neural network speech enhancement method based on attention, which is used for noise reduction of speech with background noise to realize speech enhancement. The method comprises the following steps: using a channel time-frequency attention mechanism to capture the speech distribution in the time-frequency domain; using a gated time convolution network (GTCN) to perform time modeling, so as to reduce the parameters of the network and improve the training speed of the network; and using sub-pixel convolution to realize the reconstruction of pure speech and avoid the chessboard artifacts caused by deconvolution.

[0006] The implementation process comprises the following processing,

[0007] Step 1, processing the noisy speech and the corresponding pure speech to obtain the real part and the imaginary part of the noisy speech and the real part and the imaginary part of the pure speech corresponding to the noisy speech;

[0008] Step 2, inputting the real part and the imaginary part of the noisy speech into a complex convolutional neural network based on channel time-frequency attention, training the complex convolutional neural network, calculating the loss by using the output of the complex convolutional neural network and the real part and the imaginary part of the pure speech to update the parameters of the network; the complex convolutional neural network based on channel time-frequency attention comprises a complex encoder module, a time modeling module and a complex decoder module, a channel time-frequency attention mechanism is arranged in the complex encoder module to realize the attention mechanism combining channel, time and frequency;

[0009] Step 3, repeating step 2 to continuously optimize the complex convolutional neural network until the complex convolutional network with enhanced effect meeting the preset requirement is obtained;

[0010] Step 4, processing the untrained noisy speech to obtain the real part and the imaginary part, inputting the real part and the imaginary part into the trained complex convolutional network for enhancement, and restoring the enhanced real part and the imaginary part into pure speech.

[0011] Moreover, the complex decoder module comprises a first sub-complex decoder and a second sub-complex decoder, in the complex convolutional neural network based on channel time-frequency attention,

[0012] The real part and the imaginary part of the noisy speech are first input into the complex encoder module of the network for feature extraction to obtain the output of the complex encoder module;

[0013] Then, the output of the complex encoder module is input into the time modeling module to extract the time sequence features of the speech to obtain the output of the time modeling module;

[0014] The output of the time modeling module is input into the first sub-complex decoder and the second sub-complex decoder respectively, which are respectively used for reconstructing the real part and the imaginary part of the speech; and the complex encoder module and the first sub-complex decoder and the second sub-complex decoder are connected by a skip connection.

[0015] Moreover, the complex encoder module comprises five complex convolutional layers, which are sequentially composed of a gated linear unit, a channel time-frequency attention mechanism, a layer normalization and an ELU activation, and are used for feature extraction of the processed noisy speech.

[0016] Moreover, the time modeling module is composed of a gated time convolutional network GTCN, and the gated time convolutional network GTCN comprises a two-dimensional convolution and a time convolutional network TCN, the two-dimensional convolution combines with the gating mechanism to control the information flow of the whole network, and the time convolutional network TCN retains more long-term memory related to the speech.

[0017] Moreover, the first and second sub-complex number decoder respectively comprises five layers of complex number deconvolution layers, which are sequentially composed of sub-pixel convolution, layer normalization and ELU activation, and are respectively used for estimating the real and imaginary parts of the speech spectrum.

[0018] Moreover, the channel-time-frequency attention mechanism comprises three parallel attention branches, namely channel attention, time attention and frequency attention, which guide the model to focus on the channels, time windows and features that play an important role in the speech enhancement task.

[0019] The channel-time-frequency attention mechanism comprises three parallel attention branches, namely channel attention, time attention and frequency attention, which guide the model to focus on the channels, time windows and features that play an important role in the speech enhancement task.

[0020] In another aspect, the application also provides an attention-based complex convolutional neural network speech enhancement system for implementing the above-mentioned attention-based complex convolutional neural network speech enhancement method.

[0021] Moreover, the system comprises the following modules,

[0022] The first module is used for processing the noisy speech and the corresponding pure speech to obtain the real and imaginary parts of the noisy speech and the real and imaginary parts of the corresponding pure speech of the noisy speech;

[0023] The second module is used for inputting the real and imaginary parts of the noisy speech into the complex convolutional neural network based on channel-time-frequency attention, training the complex convolutional neural network, and updating the parameters of the network by calculating the loss between the output of the complex convolutional neural network and the real and imaginary parts of the pure speech; the complex convolutional neural network based on channel-time-frequency attention comprises a complex encoder module, a time modeling module and a complex decoder module, and the channel-time-frequency attention mechanism is arranged in the complex encoder module to realize the attention mechanism combining channel, time and frequency;

[0024] The third module is used for repeatedly processing the second module to continuously optimize the complex convolutional neural network until the complex convolutional network with the enhancement effect meeting the preset requirement is obtained.

[0025] The fourth module is used for processing the untrained noisy speech to obtain the real and imaginary parts, inputting the real and imaginary parts into the trained complex convolutional network for enhancement, and restoring the enhanced real and imaginary parts into pure speech.

[0026] Alternatively, the system comprises a processor and a memory, the memory is used for storing program instructions, and the processor is used for calling the program instructions in the memory to execute the above-mentioned attention-based complex convolutional neural network speech enhancement method.

[0027] Alternatively, a readable storage medium is included, and the readable storage medium stores a computer program, and the computer program implements the attention-based complex convolutional neural network speech enhancement method when executed.

[0028] The complex convolutional neural network speech enhancement method based on channel and spatial attention provided by the application can reduce noise for speech with background noise to enhance the speech, use a channel time-frequency attention mechanism to capture the speech distribution in the time-frequency domain, use a gated time convolutional network (GTCN) for time modeling to reduce the parameters of the network and improve the training speed of the network, and use sub-pixel convolution to reconstruct pure speech to avoid chessboard artifacts caused by deconvolution. The application effectively improves the noise reduction effect of the network, reduces the noise influence in the speech signal, and thus improves the intelligibility and quality of the speech.

[0029] In addition, the application has reliable design principles, simple structure, very wide application prospects, and important market value. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 The method flowchart of the embodiments of the application is shown in the figure;

[0031] Figure 2 The complex convolutional neural network structure diagram based on channel time-frequency attention of the embodiments of the application is shown in the figure;

[0032] Figure 3 The complex convolutional layer structure diagram of the embodiments of the application is shown in the figure;

[0033] Figure 4 The complex deconvolution layer structure diagram of the embodiments of the application is shown in the figure;

[0034] Figure 5 The channel time-frequency attention mechanism diagram of the embodiments of the application is shown in the figure;

[0035] Figure 6 The time modeling module structure flowchart of the embodiments of the application is shown in the figure;

[0036] Figure 7 The time convolutional network structure flowchart of the embodiments of the application is shown in the figure. DETAILED DESCRIPTION

[0037] The technical solutions of the application are specifically described below in combination with the drawings and embodiments.

[0038] Reference Figure 1 The attention-based complex convolutional neural network speech enhancement method proposed in the embodiments of the application includes the following steps:

[0039] Step 1, the noisy speech and its corresponding pure speech are processed to obtain the real part and the imaginary part of the noisy speech and its corresponding pure speech as the training set of the network;

[0040] In specific implementation, the noisy speech and its corresponding pure speech are pre-processed by frame and window, and then the real part and the imaginary part of the noisy speech and the real part and the imaginary part of the corresponding pure speech, i.e. the training set of the network, are obtained by using short-time Fourier transform.

[0041] The application further provides that the complex convolutional neural network based on channel time-frequency attention includes a complex encoder module 1, a time modeling module 2 and a complex decoder module 3.

[0042] In the embodiment, the complex convolutional neural network based on channel time-frequency attention is specifically implemented as follows:

[0043] Referring to Figure 2 The real part and the imaginary part of the noisy speech are first input into the complex encoder module 1 of the neural network for feature extraction to obtain the output of the complex encoder module 1; then the output of the complex encoder module 1 is input into the time modeling module 2 to extract the time sequence features of the speech to obtain the output of the time modeling module 2; and then the output of the time modeling module 2 is input into the first sub-complex decoder 31 and the second sub-complex decoder 32 respectively for reconstructing the real part and the imaginary part of the speech, and the complex encoder module 1 and the first sub-complex decoder 31 and the second sub-complex decoder 32 are connected by a skip connection. The complex convolutional neural network based on channel time-frequency attention provided by the application models the long-term dependence relationship of the speech by using the time sequence of the speech signal, and has a good enhancement effect on the speech containing noise, especially for the enhancement of new speakers and non-stationary noise.

[0044] Preferably, the complex encoder module 1 includes five complex convolutional layers 11-15 connected in sequence, and the effect is better. The complex convolutional layers are sequentially composed of a gated linear unit 111, a channel time-frequency attention mechanism 112, a layer normalization 113 and an ELU activation 114, and are used for feature extraction of the processed noisy speech.

[0045] The time modeling module 2 is composed of a gated time convolution network (GTCN) including two-dimensional convolution 211-214, sigmoid activation 221-222 and time convolution network (TCN) 231-234, which extracts the timing features of the speech signal from the features output by the complex encoder module 1. Through the gated time convolution network (GTCN), a time series model is constructed from the features, retaining more long-term time-frequency features, effectively reducing the parameters of the network, improving the training speed of the network, and enabling tracking of speaker-dependent information in difficult noisy conditions. The gated time convolution network (GTCN) transmits useful information and forgets irrelevant information through the gating mechanism, thereby improving the modeling capability of the timing information, and using the time convolution network (TCN) 231-234 to retain more long-term time-frequency features of the speech.

[0046] The complex decoder module 3 is composed of two sub-decoder modules 31 and 32 with the same structure, each including five complex deconvolution layers 315-311 and 325-321 connected in turn, each complex deconvolution layer being composed of sub-pixel convolution 3111, layer normalization 3112 and ELU activation 3113 in turn, and estimating the real and imaginary parts of the speech spectrum, respectively.

[0047] The five-layer complex convolution layers 11-15 and the five-layer complex deconvolution layers 315-311 and 325-321 are connected by skip connections to alleviate the problem of gradient explosion or gradient disappearance in the network. Referring to Figure 2 The complex encoder module 1 includes complex convolution layers 11, 12, 13, 14 and 15 in turn, the sub-complex decoder 31 includes complex deconvolution layers 315-311 in turn, the sub-complex decoder 32 includes complex deconvolution layers 325-321 in turn, the first complex convolution layer 11 in the complex encoder module 1 is connected by skip connection between the output and the input of the complex deconvolution layer 311 of the sub-complex decoder 31, the second complex convolution layer 12 in the complex encoder module 1 is connected by skip connection between the output and the input of the complex deconvolution layer 312 of the sub-complex decoder 31, and so on. The fifth complex convolution layer 15 in the complex encoder module 1 is connected by skip connection between the output and the input of the complex deconvolution layer 315 of the sub-complex decoder 31; the first complex convolution layer 11 in the complex encoder module 1 is connected by skip connection between the output and the input of the complex deconvolution layer 321 of the sub-complex decoder 32, the second complex convolution layer 12 in the complex encoder module 1 is connected by skip connection between the output and the input of the complex deconvolution layer 322 of the sub-complex decoder 32, and so on. The fifth complex convolution layer 15 in the complex encoder module 1 is connected by skip connection between the output and the input of the complex deconvolution layer 325 of the sub-complex decoder 32.

[0048] The channel time-frequency attention mechanism 112 is composed of three parallel attention branches, namely channel attention 1121, time attention 1122 and frequency attention 1123, which guide the model to pay attention to which channel, which time frame and what frequency in the time frame.

[0049] When step 2 is performed for the first time, the neural network parameters can be randomly generated, and the optimization is adjusted in subsequent iterative optimization. In specific implementation, each time step 2 is performed, a fixed number of real parts and imaginary parts of the noisy speech are randomly taken from the training set for the current iteration training.

[0050] Step 3, adjust the neural network parameters (it is recommended to adjust the weights and biases of each layer of the network according to the loss of training), repeat step 2, and continuously optimize the neural network through iteration until a network with better enhancement effect is obtained, and then stop repeating iteration and enter step 4; in specific implementation, the conditions for stopping iteration can be set in advance.

[0051] Step 4, the untrained noisy speech is processed in the same way as step 1 (framing, windowing, short-time Fourier transform) to obtain the real part and imaginary part, which is input into the trained network for enhancement; the enhanced real part and imaginary part are restored into pure speech.

[0052] The overall framework of the complex convolutional neural network based on channel time-frequency attention proposed in the embodiment of the application is as shown in Figure 2 The network based on the network performs the following steps:

[0053] 1. Process the noisy speech and its corresponding pure speech to obtain the real part and imaginary part of the noisy speech and the pure speech. Wherein, the short-time Fourier transform (STFT) is used to convert the time-domain signal into the frequency-domain signal to obtain the amplitude and phase of the frequency, that is, the real part and imaginary part of the complex matrix obtained after the short-time Fourier transform.

[0054] 2. Input the real part and imaginary part of the noisy speech into the complex convolutional neural network based on channel time-frequency attention to train the network model. As shown in Figure 2 The neural network includes a complex encoder module 1, a time modeling module 2 and a complex decoder module 3.

[0055] Referring to Figure 3In the embodiment, the complex encoder module 1 comprises five complex convolution layers 11-15, each of which is sequentially composed of a gated linear unit 111, a channel time-frequency attention mechanism 112 (CTFA), layer normalization 113 and ELU activation 114. The real part and the imaginary part of the noisy speech are input into the first complex convolution layer 11 of the network as two different channels, and then sequentially pass through the gated linear unit 111 and the channel time-frequency attention mechanism (CTFA) 112 for feature extraction, layer normalization 113 and ELU activation 114 to obtain the output of the layer. The output of each layer of the network is sequentially input into the next layer for five times of complex convolution.

[0056] The gated linear unit 111 uses convolution operation to compress all information before the current time in the time window, retains the timing characteristics of the speech signal, and accelerates the operation speed through a parallel processing structure. The specific formula is as follows:

[0057]

[0058] In the formula, x represents the input of the network of the layer, y represents the output of the network of the layer, W1 and W2 are convolution kernel parameters, b1 and b2 are bias parameters, sigma() represents a sigmoid function, * represents convolution operation, represents element-wise multiplication.

[0059] The channel time-frequency attention mechanism 112 (CTFA) proposed in the application is a kind of attention mechanism module combining channel, time and frequency. Since the speech spectrum contains rich time-frequency information, the energy distribution of speech in the low frequency region is dense, and the distribution in the high frequency region is sparse, so different weights are needed to distinguish the attention degree of different regions. The CTFA model can capture the speech distribution in the time-frequency domain, suppress irrelevant background regions in the input feature map, and highlight significant features useful for specific tasks. See Figure 5 The channel time-frequency attention mechanism 112 is composed of three parallel attention branches, namely channel attention 1121, time attention 1122 and frequency attention 1123, which guide the model to pay attention to which channel, which time frame and what frequency in the time frame. The input of the module is the output of the convolution layer. In the embodiment, the process of the channel time-frequency attention mechanism 12 is as follows:

[0060] First, a one-dimensional channel attention map is generated according to the inter-channel relationship of the features. Each channel attention mechanism is constructed to establish the correlation between different features and improve the role of the convolution kernel in feature extraction under different noises. In order to effectively calculate the channel attention, the average pool and the maximum pool are used to compress the spatial dimension of the input feature to learn the correlation between channels. Let and respectively. The average-pooled and max-pooled features are forwarded to a shared network (MLP) to generate a channel attention map M c ∈R C×1×1 The calculation of channel attention is as follows:

[0061]

[0062] where F denotes the input of channel-time-frequency attention, σ() denotes the sigmoid function, MLP() denotes the shared network, AvgPool() denotes the average pooling, MaxPool() denotes the max pooling, W0∈R C / r×C and W1∈R C×C / r W0 and W1 are the weights of the MLP, C denotes the number of channels, and r denotes the dimension reduction coefficient.

[0063] Secondly, the relationship between time frames is obtained by using the time-domain features of the speech signal to generate a one-dimensional time attention map. The channel dimension and the frequency dimension of the input features are compressed using average pooling and max pooling to generate two one-dimensional feature maps: and Then a convolution operation is performed to obtain a time attention feature map, and the calculation of time attention is as follows:

[0064]

[0065] where σ() is the sigmoid function, f 7×7 () is a 7x7 size convolution kernel, AvgPool() denotes the average pooling, MaxPool() denotes the max pooling, and H denotes the height of the feature map.

[0066] Thirdly, a one-dimensional frequency attention map is generated by using the frequency-domain features of the speech signal. The channel dimension and the time dimension of the input features are compressed using average pooling and max pooling to generate two one-dimensional feature maps: and Then a convolution operation is performed to obtain a frequency attention feature map, and the calculation of frequency attention is as follows:

[0067]

[0068] where σ() is the sigmoid function, f 7×7 () is a 7x7 size convolution kernel, AvgPool() denotes the average pooling, MaxPool() denotes the max pooling, and W denotes the width of the feature map.

[0069] After that, the one-dimensional attention maps obtained from the three attention branches are multiplied to produce a final three-dimensional channel-time-frequency attention map (CTFA). The calculation is as follows:

[0070] CTFA = M c (F)*M t (F)*M f (F)

[0071] where * denotes element-wise multiplication, M c denotes channel attention map, M t denotes time attention map, M f denotes frequency attention map.

[0072] Finally, the output of the channel-time-frequency attention mechanism 12 is given by:

[0073] Y' = Y * CTFA(Y) + Y

[0074] where Y is the input of the channel-time-frequency attention mechanism, Y' is the output of the channel-time-frequency attention mechanism, CTFA() denotes the channel-time-frequency attention map, and * denotes element-wise multiplication.

[0075] The ELU activation 14 function is given by:

[0076]

[0077] where x is the input of the ELU activation function, a is an adjustable parameter, and e is the mathematical constant.

[0078] The gating temporal convolutional network (GTCN) proposed by the present application is used as the time modeling module 2 to extract more time-frequency domain features of the speech signal from the features output by the complex encoder module 1. The gating temporal convolutional network (GTCN) uses a gating mechanism to control the information flow of the entire network and uses a temporal convolutional network (TCN) 231-234 to retain more long-term memory. Compared with common speech enhancement networks such as long short-term memory network (LSTM) and recurrent neural network (RNN), the gating temporal convolutional network (GTCN) proposed by the present application has better long-time dependence capture and fewer network parameters and faster training speed. The gating temporal convolutional network (GTCN) includes two-dimensional convolution 211-214, Sigmoid activation 221-222, and temporal convolutional network (TCN) 231-234,

[0079] The gating time convolution network (GTCN) includes two-dimensional convolutions 211-214, wherein the two-dimensional convolutions 211 and 213 are the same (the number of channels, the size of the convolution kernel, and the step are the same), a smaller convolution kernel and step are set, and in the embodiment, the convolution kernel and the step are set to (1, 1) to capture local features; the two-dimensional convolutions 212 and 214 are the same (the number of channels, the size of the convolution kernel, and the step are the same), a larger convolution kernel and step are set, and in the embodiment, the convolution kernel is set to (3, 3) and the step is set to (1, 2) to extract high-level features; the output of each two-dimensional convolution is scaled to (0, 1) through Sigmoid activation 221, 222, and multiplied by the output of another two-dimensional convolution in the same layer, see Figure 6 , the output of the two-dimensional convolution 211 is multiplied element by element with the output after the Sigmoid activation 221 of the two-dimensional convolution 212, the output of the two-dimensional convolution 212 is multiplied element by element with the output after the Sigmoid activation 221 of the two-dimensional convolution 211, and the multiplied results are added element by element and sent to the time convolution network (TCN) 231-234; the output of the two-dimensional convolution 213 is multiplied element by element with the output after the Sigmoid activation 222 of the two-dimensional convolution 214, the output of the two-dimensional convolution 214 is multiplied element by element with the output after the Sigmoid activation 222 of the two-dimensional convolution 213, and the multiplied results are added element by element and sent to the complex decoder 3 as the output of the time modeling module 2.

[0080] The time convolution network (TCN) 231-234 is a convolution network proposed for a sequence modeling task with causal constraints. The TCN can take a sequence of arbitrary length and map it to an output sequence of the same length. The time convolution network (TCN) 231-234 is connected in sequence, and the structures of the four time convolution networks (TCN) are the same, only the dilation rates of the two-dimensional dilated convolutions are different, and the dilation rates increase in turn. In the embodiment, the dilation rates are 1, 2, 4, and 8.

[0081] Referring to Figure 7 , the time convolution network (TCN) 231 sequentially includes a two-dimensional dilated convolution 2311, weight normalization 2312, ReLU activation 2313, Dropout 2314, a two-dimensional dilated convolution 2315, weight normalization 2316, ReLU activation 2317, Dropout 2318, and the input of the time convolution network (TCN) 231 is connected in residual connection with the output of the Dropout 238.

[0082] The expression of the two-dimensional dilated convolution is:

[0083] Y = (X * F d )

[0084] In the formula, X and Y represent input features and output features, d represents the expansion rate, and F represents the expanded convolution kernel for a convolution kernel of size k*k d of size k d *k d , wherein

[0085] k d = k + (k - 1) (r - 1)

[0086] The complex decoder module 3 is composed of two sub-decoder modules 31 and 32 with the same structure, each of which includes five complex deconvolution layers 315-311 and 325-321, as shown in Figure 4 Each complex deconvolution layer is composed of a sub-pixel convolution 3111, layer normalization 3112 and ELU activation 3113 in sequence, and estimates the real and imaginary parts of the speech spectrum, respectively.

[0087] The sub-pixel convolution 3111 is a method for recovering high-resolution pictures and videos from low-resolution in digital image processing, which can increase the height and width of the feature map after convolution. The sub-pixel convolution can be used instead of deconvolution to realize the reconstruction of clean speech, avoiding the chessboard artifacts caused by deconvolution. The calculation formula is as follows:

[0088] y = PS (W * x + b)

[0089] In the formula, x represents the input of the current layer network, y represents the output of the current layer network, W is the convolution kernel parameter, b is the bias parameter, * represents the convolution operation, and PS() represents the PixelShuffle operation (a kind of up-sampling module, which can obtain a high-resolution feature map of a specified magnification by rearranging the pixel points of the feature map of multiple channels in rows and columns through setting the up-sampling ratio).

[0090] The complex convolution layer in the complex encoder module 1 and the complex deconvolution layer in the complex decoder module 3 are connected by a skip connection, and the result after connection is used as the input of the next complex deconvolution layer. This is used to alleviate the problem of gradient explosion or gradient disappearance of the network. The formula is as follows:

[0091] y″ i+1 = y i + y′ i

[0092] In the formula, y″ i+1 represents the input of the (i+1)th complex deconvolution layer, y i represents the output of the ith complex convolution layer, y′ i represents the output of the ith complex deconvolution layer.

[0093] 3. The parameters of the network model are updated by calculating the loss of the output of the neural network and the real and imaginary parts of the corresponding pure speech. The network is repeatedly trained and the loss is calculated to update the parameters of the network model, and the neural network is continuously optimized until a network with better enhancement effect is obtained. When calculating the error Loss, the mean square error is used, and the specific calculation formula is as follows:

[0094]

[0095] In the formula, n represents the number of elements, f(x) represents the real and imaginary parts after network enhancement, and y represents the real and imaginary parts of the pure speech.

[0096] 4. The untrained noisy speech is processed by short-time Fourier transform to obtain the real and imaginary parts, which are input into the trained network for enhancement. The enhanced real and imaginary parts are restored into pure speech by inverse short-time Fourier transform.

[0097] In specific implementation, the method proposed in the technical scheme of the present application can be automatically run by computer software technology, and the system device of the method, such as a computer readable storage medium storing the corresponding computer program of the technical scheme of the present application and a computer device including the running of the corresponding computer program, should also be within the protection scope of the present application.

[0098] In some possible embodiments, a complex convolutional neural network speech enhancement system based on attention is provided, including the following modules,

[0099] The first module is used for processing noisy speech and corresponding pure speech to obtain the real and imaginary parts of the noisy speech and the real and imaginary parts of the corresponding pure speech of the noisy speech;

[0100] The second module is used for inputting the real and imaginary parts of the noisy speech into a complex convolutional neural network based on channel time-frequency attention, training the complex convolutional neural network, calculating the loss of the output of the complex convolutional neural network and the real and imaginary parts of the pure speech, and updating the parameters of the network. The complex convolutional neural network based on channel time-frequency attention includes a complex encoder module, a time modeling module and a complex decoder module, and a channel time-frequency attention mechanism is arranged in the complex encoder module to realize the attention mechanism combining channels, time and frequency.

[0101] The third module is used for repeatedly processing the second module to continuously optimize the complex convolutional neural network until a complex convolutional network with an enhancement effect meeting the preset requirements is obtained.

[0102] The fourth module is used for processing untrained noisy speech to obtain the real and imaginary parts, inputting the real and imaginary parts into the trained complex convolutional network for enhancement, and restoring the enhanced real and imaginary parts into pure speech.

[0103] In some possible embodiments, there is provided an attention-based complex convolutional neural network speech enhancement system, comprising a processor and a memory, the memory being configured to store program instructions, and the processor being configured to invoke the program instructions in the memory to perform an attention-based complex convolutional neural network speech enhancement method as described above.

[0104] In some possible embodiments, there is provided an attention-based complex convolutional neural network speech enhancement system, comprising a readable storage medium, and a computer program stored on the readable storage medium, the computer program being configured to implement an attention-based complex convolutional neural network speech enhancement method as described above when executed.

[0105] The specific embodiments described herein are merely illustrative of the spirit of the application. Various modifications or changes in addition or substitution to the described specific embodiments can be made by those skilled in the art without departing from the spirit of the application or exceeding the scope of the appended claims.

Claims

1. An attention-based complex convolutional neural network speech enhancement method, characterized in that: The method comprises the following steps: (1) processing the noisy speech and the corresponding clean speech to obtain the real part and the imaginary part of the noisy speech and the real part and the imaginary part of the clean speech corresponding to the noisy speech; (2) inputting the real part and the imaginary part of the noisy speech into a complex convolutional neural network based on channel time-frequency attention, training the complex convolutional neural network, calculating the loss by using the output of the complex convolutional neural network and the real part and the imaginary part of the clean speech, and updating the parameters of the network; the complex convolutional neural network based on channel time-frequency attention comprises a complex encoder module, a time modeling module and a complex decoder module, and the channel time-frequency attention mechanism is arranged in the complex encoder module to realize the attention mechanism combining channel, time and frequency; the complex encoder module preferably comprises five layers of complex convolutional layers connected in sequence, and each complex convolutional layer is composed of a gated linear unit, a channel time-frequency attention mechanism, a layer normalization and an ELU activation in sequence; the time modeling module is composed of a gated time convolutional network GTCN, which comprises a two-dimensional convolution, a sigmoid activation and a time convolutional network TCN, extracts the time sequence features of the speech signal from the features output by the complex encoder module, constructs a time sequence model from the features by using the gated time convolutional network GTCN, retains more long-term time-frequency features, reduces the parameters of the network, improves the training speed of the network, and supports tracking speaker-dependent information in difficult noisy conditions; the gated time convolutional network GTCN transmits useful information and forgets irrelevant information through the gating mechanism, thereby improving the modeling capability of the time sequence information, and using the time convolutional network TCN to retain more long-term time-frequency features of the speech; the complex decoder module is composed of two sub-decoder modules with the same structure, which respectively estimate the real part and the imaginary part of the speech spectrum; each sub-decoder module comprises five layers of complex deconvolutional layers connected in sequence, and each complex deconvolutional layer is composed of a sub-pixel convolution, a layer normalization and an ELU activation in sequence; (3) repeating step 2 to continuously optimize the complex convolutional neural network until the complex convolutional network with the enhanced effect meeting the preset requirements is obtained; (4) processing the untrained noisy speech to obtain the real part and the imaginary part, and inputting the real part and the imaginary part into the trained complex convolutional network for enhancement; recovering the enhanced real part and imaginary part into clean speech.

2. The complex convolutional neural network speech enhancement method based on attention according to claim 1, wherein: the complex decoder module comprises a first sub-complex decoder and a second sub-complex decoder, and in the complex convolutional neural network based on channel time-frequency attention, the real part and the imaginary part of the noisy speech are first input into the complex encoder module of the network for feature extraction to obtain the output of the complex encoder module. ​ The output of the complex encoder module is input into a time modeling module, and time sequence features of the speech are extracted to obtain an output of the time modeling module. The output of the time modeling module is input into a first sub-complex decoder and a second sub-complex decoder respectively, and the first sub-complex decoder and the second sub-complex decoder are used to reconstruct the real part and the imaginary part of the speech respectively.

3. The attention-based complex convolutional neural network speech enhancement method according to claim 1, characterized in that: The complex encoder module includes five complex convolution layers, which are sequentially composed of a gated linear unit, a channel time-frequency attention mechanism, layer normalization and ELU activation, and are used to extract features of the processed noisy speech.

4. The attention-based complex convolutional neural network speech enhancement method according to claim 1, characterized in that: The time modeling module is composed of a gated time convolution network (GTCN), which includes a two-dimensional convolution and a time convolution network (TCN). The two-dimensional convolution combines a gating mechanism to control the information flow of the entire network, and the time convolution network (TCN) retains more long-term memory related to the speech.

5. The attention-based complex convolutional neural network speech enhancement method according to claim 1, characterized in that: The first sub-complex decoder and the second sub-complex decoder each include five complex deconvolution layers, which are sequentially composed of a sub-pixel convolution, layer normalization and ELU activation, and are used to estimate the real part and the imaginary part spectrum of the speech respectively.

6. The attention-based complex convolutional neural network speech enhancement method according to claim 1 or 2 or 3 or 4 or 5, characterized in that: The channel time-frequency attention mechanism is composed of three parallel attention branches, which are channel attention, time attention and frequency attention, and guide the model to focus on the channels, time windows and features that play an important role in the speech enhancement task.

7. An attention-based complex convolutional neural network speech enhancement system, characterized in that: A complex convolutional neural network speech enhancement method based on attention is provided. The method comprises the following modules, A first module is configured to process the noisy speech and the corresponding clean speech to obtain the real part and the imaginary part of the noisy speech, and the real part and the imaginary part of the clean speech corresponding to the noisy speech. A second module is configured to input the real part and the imaginary part of the noisy speech into a complex convolutional neural network based on channel time-frequency attention, train the complex convolutional neural network, and update the parameters of the network by calculating the loss between the output of the complex convolutional neural network and the real part and the imaginary part of the clean speech. The complex convolutional neural network based on channel time-frequency attention includes a complex encoder module, a time modeling module and a complex decoder module. The channel time-frequency attention mechanism is arranged in the complex encoder module to realize the attention mechanism combining channel, time and frequency. A third module is configured to repeatedly process the second module to continuously optimize the complex convolutional neural network until a complex convolutional network with a satisfactory enhancement effect is obtained. A fourth module is configured to process the untrained noisy speech to obtain the real part and the imaginary part, and input the real part and the imaginary part into the trained complex convolutional network for enhancement. The enhanced real part and imaginary part are restored into clean speech.

8. The attention-based complex convolutional neural network speech enhancement system of claim 7, wherein: A processor and a memory are provided. The memory is configured to store program instructions, and the processor is configured to call the program instructions in the memory to execute the complex convolutional neural network speech enhancement method based on attention. 9.The attention-based complex convolutional neural network speech enhancement system of claim 7, wherein: The application also discloses a readable storage medium, and the readable storage medium stores a computer program. The computer program is used for implementing the attention-based complex convolutional neural network speech enhancement method according to any one of claims 1-6.

Images