Recording and broadcasting classroom speech enhancement method based on deep learning

Abstract

The invention discloses a recording and broadcasting classroom voice enhancement method based on deep learning, which is used for enhancing voice signals acquired in a recording and broadcasting classroom so as to improve the audio definition and audio information intelligibility of the recording and broadcasting classroom. The invention provides an improved network structure model based on a time-frequency long-short time memory network (F-T-LSTM) and a mixed attention mechanism on the basis of a traditional deep convolutional neural network. The model is improved on the basis of a deep re-convolution recursive network, the correlation between a voice time domain and a voice frequency domain is described more accurately by introducing an F-T-LSTM structure, a channel attention mechanism and a space attention mechanism are fused to form a mixed attention mechanism, more comprehensive feature extraction is performed on voice features of a complex domain, and the accuracy of voice feature extraction is improved. Calculation force is ingeniously concentrated in the field with the most abundant speech spectrogram feature information, so that the overall performance of the speech enhancement network in the complex field is remarkably improved, and the network model has good generalization ability.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and speech signal processing, and relates to a speech enhancement technology based on a recording and broadcasting classroom, and specifically to a speech enhancement method for a recording and broadcasting classroom based on a deep learning algorithm. Background Art

[0002] In daily life, people convey information through voice, which is a widespread communication medium. With the development of artificial intelligence, voice communication is not limited to people, but also exists between people and machine equipment. The development and application of machine hearing systems have become an important research direction. In a complex acoustic environment, environmental sound interference, room reverberation, and interference from other speakers will affect the quality of voice audio collected by machine equipment, thereby affecting the recognition and analysis of voice signals. Especially in actual environments, noise signals, reverberation signals and pure voice signals are difficult to separate, so improving the quality of voice signals for intelligent voice devices has long been a research goal.

[0003] Nowadays, with the development of communication technology and the popularization of information-based teaching in campus teaching, people have put forward new standards and new requirements for high-quality information-based teaching classrooms. Especially under the background of the "Education Informatization 2.0 Action Plan", many colleges and universities have improved the level of education informatization by building campus smart classrooms, among which the creation of regular recording and broadcasting classrooms has become an important informatization construction scenario. A large amount of classroom recording and broadcasting audio and video resources can not only be used by teachers and students for teaching activities, but these course resources will also serve as valuable knowledge materials for schools to build professional, subject systems and knowledge graphs. However, due to various classroom environmental factors and differences in hardware equipment, many audio and video resources present noisy ambient sounds, which makes the recorded courses unable to meet the needs of teachers and students for after-class review or for intelligent teaching monitoring.

[0004] Therefore, deep learning models are used to learn the characteristics and patterns of speech signals from a large amount of speech data, and then used for speech enhancement tasks in the classroom, thereby ensuring that the audio resources have better listening effects and are more valuable, achieving maximum optimization of recorded audio, being able to hear what teachers and students are saying clearly, and achieving the effect of low ambient noise, thereby achieving a good teaching and learning experience. Summary of the invention

[0005] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for speech enhancement in a recording and broadcasting classroom based on deep learning. The purpose of the present invention can be achieved through the following technical solutions:

[0006] A method for voice enhancement in a recording and broadcasting classroom based on deep learning, the method comprising the following steps:

[0007] Step 1: Obtain sufficient real classroom speech data sets, divide the speech data sets into clean speech and noisy speech according to the noise content of the speech, and divide the data sets into training sets and test sets;

[0008] Step 2: Use the deep learning algorithm deep convolutional neural network DCCRN structure as the basic structure of this speech enhancement model.

[0009] Step 3: Improve the network structure, use the time-frequency long short-term memory network FT-LSTM to replace the T-LSTM structure, and cascade the channel attention mechanism and the spatial attention mechanism to form a hybrid attention mechanism CBAM and integrate it into the complex convolution.

[0010] Step 4: Based on the training set, train the improved deep convolutional neural network structure model: DCCPX, select parameters and test the model effect to obtain the trained enhanced network model;

[0011] Step 5: Perform speech preprocessing on the noisy speech based on the test set, input the processed speech into the trained enhancement network model, and obtain the enhanced speech output predicted by the model to achieve the purpose of speech enhancement.

[0012] Furthermore, the step 1 comprises the following sub-steps:

[0013] Step 101: Use professional audio equipment to collect voice data in real time, obtain a voice data set containing various classroom noise conditions, and synthesize it with a public classroom clean voice data set to obtain a final data set, which contains clean voice and noisy voice, and use it as training data, and use the corresponding voice content as training labels to form a sample set;

[0014] Step 102: Divide the sample set into a training set and a test set, wherein the training set includes clean speech and noisy speech, and the test set includes noisy speech that is not in the training set. Perform data preprocessing on the training set and the test set, including pre-emphasis, framing, windowing, and other operations, to ensure signal stability, smoothness, parameter extraction effect, and overall quality optimization in subsequent speech processing.

[0015] Furthermore, the structure of the base network deep complex convolutional neural network DCCRN is as follows: the number of encoder output channels is {16, 32, 64, 128, 256, 256}. A two-layer complex LSTM is used with 256 units. The number of decoder input channels is {256+256, 256+256, 128+128, 64+64, 32+32, 16+16}. During the network training process, the size and step size of the convolution kernel are (5, 2) and (2, 1) respectively.

[0016] Furthermore, the improved deep convolutional neural network structure in step 3 specifically includes:

[0017] The time-frequency long short-term memory network FT-LSTM replaces the T-LSTM structure, where the input is assumed to be the time-frequency representation of speech X∈R F×T×C , F represents the frequency dimension, T is the time dimension, and C is the number of channels (for complex input, C = 2): FT-LSTM can be expressed as:

[0018] F-LSTM:h_f(t),c_f(t)=LSTM_f(X(:,t,:),h_f(t-1),c_f(t-1))

[0019] T-LSTM:h_t(f),c_t(f)=LSTM_f(X(f,:,:),h_t(f-1),c_t(f-1))

[0020] FT-LSTM:α⊙h_f+(1-α)h_t

[0021] Where X(:,t,:) represents all frequency data at time t, X(f,:,:) represents all time data at frequency f, h_f(t), h_t(f) and c_f(t), c_t(f) represent hidden states and cell states respectively. α is a learnable weight parameter used to balance the importance of frequency domain and time domain information. ⊙ represents element-wise multiplication.

[0022] The cascaded channel attention mechanism and the spatial attention mechanism form a hybrid attention mechanism CBAM, which is integrated between the encoder and decoder of the deep convolutional neural network and after the decoder. It enhances the model's ability to pay attention to features by dynamically adjusting the speech feature image and the corresponding weight parameters.

[0023] Furthermore, the step 4 comprises the following sub-steps:

[0024] Step 401: Extract speech spectrogram features, define a suitable loss function according to task requirements: SI-SNR, and train the model using the prepared training set. The SI-SNR formula is as follows:

[0025]

[0026] in, is the evaluation signal, s represents the pure signal, is the element-wise product and sum operation, ||s‖ 2 It is L2norm (2 norm), which can be simply understood as the distance from all points in two-dimensional space to the center of the circle, which is equivalent to<s,s> .

[0027] Step 402: During the training process, different hyperparameters are adjusted for the improved deep convolutional neural network structure model and trained separately, and the weights, biases and other parameters of the model are updated through the optimization algorithm Adam to improve the performance of the model and obtain an enhanced model.

[0028] Preferably, when extracting speech data features: use a time-frequency domain long short-term memory network to model in both time and frequency dimensions to capture the features and dependencies of speech time series data in the frequency domain. Use a spatial attention mechanism and a channel attention mechanism to calculate the complementary attention of speech feature graphs, thereby extracting multi-dimensional speech features.

[0029] Beneficial effects of the present invention:

[0030] (1) The present invention introduces the time-frequency domain LSTM, which is combined with the frequency and time LSTM to more accurately describe the correlation between the time domain and frequency domain of speech and reduce the amount of calculation.

[0031] (2) The present invention integrates two attention modules, channel and space, and can be easily integrated into any complex-valued convolutional layer to further mine the multi-dimensional feature information of the speech feature graph, improve the performance of the speech enhancement system and reduce the impact of noise. It can also cleverly concentrate the computing power on the area with the richest feature data, optimize the computing efficiency of the network, and ensure that the model can be more accurate and efficient when processing complex speech signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of a specific implementation method of a recording and broadcasting classroom speech enhancement method based on deep learning of the present invention.

[0033] Figure 2 This is a diagram illustrating the training and testing phases of a deep learning-based recording classroom speech enhancement method of the present invention.

[0034] Figure 3 This is a network structure diagram of a deep learning-based voice enhancement method for a recording and broadcasting classroom according to the present invention.

[0035] Figure 4 This is a characteristic spectrogram of a deep learning-based speech enhancement method for a recording classroom in the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, but the present invention is not limited thereto.

[0037] Example:

[0038] Please refer to Figure 1-4As shown, the present invention provides a method for voice enhancement in a recording and broadcasting classroom based on deep learning, comprising the following steps:

[0039] Step 1: Obtain sufficient real classroom speech datasets and divide the datasets into training sets and test sets:

[0040] Professional audio equipment is used to collect voice data in real time, and a voice data set containing various classroom noise conditions is obtained. The data set is synthesized with the clean voice data set of the public classroom to obtain the final data set, which contains clean speech and noisy speech. The data set is used as training data, and the corresponding speech content is used as training labels to form a sample set.

[0041] The sample set is divided into a training set and a test set. The training set contains clean speech and noisy speech, and the test set contains noisy speech that is not in the training set. Data preprocessing is performed on the training set and the test set, including pre-emphasis, framing, windowing and other operations to ensure signal stability, smoothness, parameter extraction effect and overall quality optimization in subsequent speech processing.

[0042] Step 2: The deep convolutional neural network DCCRN structure of the deep learning algorithm is used as the basic structure of this speech enhancement model: the number of encoder output channels is {16, 32, 64, 128, 256, 256}. A two-layer complex LSTM is used with 256 units. The number of decoder input channels is {256+256, 256+256, 128+128, 64+64, 32+32, 16+16}. During the network training process, the size and step size of the convolution kernel are (5, 2) and (2, 1) respectively.

[0043] Step 3: Improve the network structure, use the time-frequency long short-term memory network FT-LSTM to replace the T-LSTM structure, and cascade the channel attention mechanism and the spatial attention mechanism to form a hybrid attention mechanism CBAM and integrate it into the complex convolution:

[0044] The time-frequency long short-term memory network FT-LSTM replaces the T-LSTM structure, where the input is assumed to be the time-frequency representation of speech X∈R F×T×C , F represents the frequency dimension, T is the time dimension, and C is the number of channels (for complex input, C = 2): FT-LSTM can be expressed as:

[0045] F-LSTM:h_f(t),c_f(t)=LSTM_f(X(:,t,:),h_f(t-1),c_f(t-1))

[0046] T-LSTM:h_t(f),c_t(f)=LSTM_f(X(f,:,:),h_t(f-1),c_t(f-1))

[0047] FT-LSTM:α⊙h_f+(1-α)h_t

[0048] Where X(:,t,:) represents all frequency data at time t, X(f,:,:) represents all time data at frequency f, h_f(t), h_t(f) and c_f(t), c_t(f) represent hidden states and cell states respectively. α is a learnable weight parameter used to balance the importance of frequency domain and time domain information. ⊙ represents element-wise multiplication.

[0049] The cascaded channel attention mechanism and the spatial attention mechanism form a hybrid attention mechanism CBAM, which is integrated between the encoder and decoder of the deep convolutional neural network and after the decoder. It enhances the model's ability to pay attention to features by dynamically adjusting the speech feature image and the corresponding weight parameters.

[0050] Step 4: Based on the training set, train the improved deep convolutional neural network structure model, select parameters and test the model effect to obtain the trained enhanced network model:

[0051] In the training phase, clean speech and noisy speech data sets are input to preprocess the speech data, and then speech feature information is extracted from the preprocessed speech to establish a speech enhancement model for model training. The model compares with the real clean speech signal, optimizes the loss by minimizing the difference between the predicted output and the real output, and gradually adjusts the parameters according to the corresponding model parameters to improve the processing ability of the noise signal and restore the clean speech signal as much as possible.

[0052] Step 5: Perform speech preprocessing on the noisy speech based on the test set, input the processed speech into the trained enhancement network model, and obtain the enhanced speech output predicted by the model to achieve the purpose of speech enhancement.

[0053] In this example, the software environment used is Nvidia GeForce GTX 3080Ti, based on Python 3.8 development language, Pytorch 1.11.0 deep learning framework, and Cuda 11.3.0 to build a network model for training. The model uses the Adam optimizer, the batch size is set to 8, the epoch is set to 90, and the learning rate is set to 0.001. The short-time Fourier transform length n_fft is set to 512, and the sampling rate is set to 16KHz. The audio clarity metric PESQ and the speech short-time objective intelligibility metric STOI are used to evaluate the enhancement effect. Both compare the difference between the enhanced speech and the pure speech on the test set. The larger the indicator value, the better the enhancement effect.

[0054] The method proposed in the present invention is finally tested in a synthetic speech dataset. Compared with the DCCRN basic model, PESQ is improved by 3.6% and STOI is improved by 3.1%. Under the same setting conditions, compared with Wiener, LSTM, CRN and DPCRN algorithms, PESQ is 0.474%, 0.2, 0.197 and 0.021 higher than the above four algorithms, respectively, and STOI is 8.55%, 6.69%, 6.33% and 0.52% higher than the above four algorithms, respectively. In addition, the Gflops of the improved DCCPX algorithm is lower than that of the baseline network, which shows that this method not only optimizes the computational efficiency of the network, but also ensures that the model can be more accurate and efficient when processing complex speech signals.

Claims

1. A method for speech enhancement in a recording and broadcasting classroom based on deep learning, characterized in that: The method comprises the following steps: Step 1: Obtain sufficient real classroom speech data sets, divide the speech data sets into clean speech and noisy speech according to the noise content of the speech, and process the speech data sets into a training set and a test set; Step 2: Use the deep learning algorithm deep convolutional neural network DCCRN structure as the basic structure of this speech enhancement model; Step 3: Improve the network structure, use the time-frequency long short-term memory network FT-LSTM to replace the T-LSTM structure, and cascade the channel attention mechanism and the spatial attention mechanism to form a hybrid attention mechanism CBAM and integrate it into the complex convolution; Step 4: Based on the training set, the improved deep convolutional neural network structure model DCCPX is trained, parameters are selected and the model effect is tested to obtain a trained enhanced network model; Step 5: Perform speech preprocessing on the noisy speech based on the test set, input the processed speech into the trained enhanced network model, and obtain the enhanced speech output predicted by the model to achieve the purpose of speech enhancement.

2. The method for voice enhancement in a recording and broadcasting classroom based on deep learning according to claim 1 is characterized in that: The step 1 comprises the following sub-steps: Step 101: Use professional audio equipment to collect voice data in real time, obtain a voice data set containing various classroom noise conditions, and synthesize it with a public classroom clean voice data set to obtain a final data set, which contains clean voice and noisy voice, and use it as training data, and use the corresponding voice content as training labels to form a sample set; Step 102: Divide the sample set into a training set and a test set, wherein the training set includes clean speech and noisy speech, and the test set includes noisy speech that is not in the training set, and perform data preprocessing on the training set and the test set, including: pre-emphasis, framing, and windowing operations, to ensure signal stability, smoothness, parameter extraction effect, and overall quality optimization in subsequent speech processing.

3. The method for voice enhancement in a recording and broadcasting classroom based on deep learning according to claim 1 is characterized in that: In step 2, the network DCCRN structure model is: the number of encoder output channels is {16, 32, 64, 128, 256, 256}, a two-layer complex LSTM is used, the number of units is 256, the number of decoder input channels is {256+256, 256+256, 128+128, 64+64, 32+32, 16+16}, and during the network training process, the size and step size of the convolution kernel are (5, 2) and (2, 1) respectively.

4. The method for voice enhancement in a recording and broadcasting classroom based on deep learning according to claim 1 is characterized in that: The improved deep convolutional neural network structure in step 3 specifically includes: The time-frequency long short-term memory network FT-LSTM replaces the T-LSTM structure, where the input is assumed to be the time-frequency representation of speech X∈R F×T×C , F represents the frequency dimension, T is the time dimension, and C is the number of channels (for complex input, C = 2): FT-LSTM can be expressed as: F-LSTM:h_f(t),c_f(t)=LSTM_f(X(:,t,:),h_f(t-1),c_f(t-1)) T-LSTM:h_t(f),c_t(f)=LSTM_f(X(f,:,:),h_t(f-1),c_t(f-1)) FT-LSTM:α⊙h_f+(1-α)h_t Among them, X(:,t,:) represents all frequency data at time t, X(f,:,:) represents all time data at frequency f, h_f(t), h_t(f) and c_f(t), c_t(f) represent hidden states and cell states respectively, α is a learnable weight parameter used to balance the importance of frequency domain and time domain information, and ⊙ represents element-wise multiplication. The cascaded channel attention mechanism and the spatial attention mechanism form a hybrid attention mechanism CBAM, which is integrated between the encoder and decoder of the deep convolutional neural network and after the decoder. It enhances the model's ability to pay attention to features by dynamically adjusting the speech feature image and the corresponding weight parameters.

5. The method for voice enhancement in a recording and broadcasting classroom based on deep learning according to claim 1 is characterized in that: The step 4 comprises the following sub-steps: Step 401: Extract speech spectrogram features, define a suitable loss function according to task requirements: SI-SNR, and train the model using the prepared training set. The SI-SNR formula is as follows: in, is the evaluation signal, s represents the pure signal, is the element-wise product and sum operation, ‖s‖ 2 It is L2norm (2 norm), which can be simply understood as the distance from all points in two-dimensional space to the center of the circle, which is equivalent to<s,s> ; Step 402: During the training process, different hyperparameters are adjusted for the improved deep convolutional neural network structure model and trained separately, and the weights, biases and other parameters of the model are updated through the optimization algorithm Adam to improve the performance of the model and obtain an enhanced model.

Images