Speech enhancement method based on time sequence convolutional network
This speech enhancement method based on temporal convolutional networks addresses the intelligibility problem of audio data in complex environments. Through feature extraction and temporal correlation, it significantly improves the quality and intelligibility of audio data and is applicable to various speech enhancement tasks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV OF POSTS & TELECOMM
- Filing Date
- 2026-02-09
- Publication Date
- 2026-05-26
AI Technical Summary
Existing audio enhancement algorithms cannot quickly and effectively acquire clean audio information in complex external interference environments.
A speech enhancement method based on temporal convolutional networks is adopted. Audio feature values are extracted by convolutional short-time Fourier transform, and data smoothing and correlation are performed by combining one-dimensional convolution and temporal convolutional networks. Causal and extended causal convolutional layers are constructed, activation functions and dropout mechanisms are introduced, and temporal convolutional modules are combined for training and decoding. The effect is evaluated using a variety of objective evaluation metrics.
It significantly improves the intelligibility of audio data, enhances speech enhancement performance in complex environments, and the model is flexible and adaptable, suitable for various speech enhancement tasks.
Smart Images
Figure CN122090855A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of electronic information applications, specifically a speech enhancement method based on temporal convolutional networks. Background Technology
[0002] In reality, acquiring effective audio data is often accompanied by interference from venue, equipment, or other external factors, posing a significant challenge to the intelligibility of audio data. To address this issue, various audio enhancement methods have emerged and developed. Traditional methods using adaptive filtering algorithms to enhance speech are no longer sufficient for rapidly extracting effective information from audio data in large-scale, complex, and ever-changing background environments. Artificial intelligence (AI) has effectively solved this problem. Therefore, in recent years, methods using AI to enhance speech quality have achieved significant results. Among these, methods utilizing neural networks for large-scale speech enhancement are particularly outstanding. Convolutional neural networks can effectively extract the correlation between data, while temporal neural networks can effectively capture the correlation between data and time changes. Combining these two methods can efficiently obtain the spatial and temporal relationships of data, thereby improving the model's performance in complex scenarios. Applying this algorithm to speech enhancement can effectively distinguish between effective information and invalid noise in audio. By removing noise and enhancing effective information, the quality of audio data is improved.
[0003] Speech enhancement methods based on temporal convolutional networks combine techniques such as frequency domain transformation, feature extraction, and deep learning. By analyzing large amounts of audio signal data, short-time Fourier transform and one-dimensional convolutions are used to convert the signal data into frequency domain data, obtaining and smoothing feature values. Temporal convolutional networks are then used to extract the spatial and temporal relationships between the data, performing correlation and inference to extract effective information and avoid irrelevant noise. This data enhancement process improves the intelligibility of the audio data. This method can improve the effectiveness and real-time performance of audio data enhancement, quickly and efficiently helping audio receivers understand the effective information contained in the audio. Summary of the Invention
[0004] This invention aims to address the problem that existing audio enhancement algorithms cannot quickly and effectively acquire clean audio information in situations with a large amount of complex external interference. Therefore, this invention proposes a speech enhancement method based on temporal convolutional networks. Specifically, it includes the following steps:
[0005] Acquire the dataset and preprocess the data to generate data pairs for training the model.
[0006] Obtain the datasets needed to train the model, including clean audio data, noise audio data, room impulse response audio data (RIR), etc.
[0007] Clean data is randomly mixed with noisy data and RIR data to generate an audio dataset with background noise that closely resembles what might happen in reality. The original clean audio data is used as the target value, and the generated data pairs are used for subsequent training.
[0008] Based on general industry knowledge rules and the needs of actual application scenarios, construct the system's measurement standards.
[0009] Based on industry-standard knowledge and rules, commonly used metrics for measuring audio quality are selected as some evaluation indicators for evaluating speech enhancement effects.
[0010] To meet the needs of real-world application scenarios, representative audio quality metrics were selected and combined with the aforementioned general metrics to construct a comprehensive audio quality measurement system.
[0011] An encoder is constructed using Convolutional Short-Time Fourier Transform (ConvSTFT) as the feature value of the audio data, which is then used as the input term of the model.
[0012] The short-time Fourier transform of the audio data is calculated to extract the desired audio information as feature values. The calculation formula is as follows:
[0013] (1),
[0014] Where w(t) is the window function, usually a Hanning window or a Gaussian window centered at zero, and x(t) is the signal to be transformed. For time shift parameters, Angular frequency parameter, This is the time-frequency representation of the short-time Fourier transform of the audio data.
[0015] One-dimensional convolution is introduced to smoothly process audio information into input terms that fit the neural network.
[0016] A temporal convolutional network (TCN) is introduced to train the model, which then decodes the encoder to obtain the speech enhancement result.
[0017] The formula for constructing a causal convolutional layer is as follows:
[0018] (2),
[0019] Where F is the filter, X is a sequence. , The kernel size is the convolution kernel size. for The values of the X sequence at time t are given by the formula above. The causal convolution value at the location;
[0020] The above formula represents in The causal convolution value at that location.
[0021] Construct extended causal convolutional layers by increasing the number of intervals in the convolutional kernels, i.e., setting the dilation size. The number of intervals increases with the number of layers, and its calculation formula is as follows:
[0022] (3),
[0023] Where F is the filter, X is a sequence. d is the expansion dimension. The kernel size is the convolution kernel size. for The values of the X sequence at time t are given by the formula above. The causal convolution value at the location;
[0024] The above formula represents in The causal convolution value with an expansion size of d at the point.
[0025] The weights are normalized, and the formula for calculation is as follows:
[0026] (4),
[0027] Where g is a scalar of control amplitude and v is a unit vector of control direction.
[0028] The activation function ReLU is introduced, and its calculation formula is as follows:
[0029] (5),
[0030] Dropout is introduced to prevent network layers from overfitting to the audio data used for training. Dropout is a regularization layer that randomly discards the output of neurons with a forward probability p, introducing randomness during training and forcing the network to learn more robustly.
[0031] The above algorithms are combined into a temporal convolution module.
[0032] Multiple temporal convolutional modules are combined to form a temporal convolutional network, which is used to train the model and obtain audio temporal information. By using multiple convolutional kernels with increasing dilation rates in parallel, multi-scale temporal features are extracted and fused within a single module.
[0033] The encoder is decoded to obtain the enhanced audio signal data.
[0034] Three objective evaluation metrics were primarily selected as the assessment standards for speech quality: Scale-Invariant Signal-to-Noise Ratio (SISNR), Perceptual Speech Quality Assessment (PESQ), and Short-Time Objective Intelligibility (STOI). SISNR primarily assesses the waveform similarity between the enhanced audio data and the clean speech audio data; a higher value indicates greater similarity, meaning a better speech enhancement effect. PESQ primarily assesses the overall speech quality after enhancement; its output range is typically between -0.5 and 4.5, with higher scores indicating better overall speech quality. STOI focuses on assessing the intelligibility of the speech audio, typically ranging from 0 to 1; a value closer to 1 indicates higher intelligibility, thus analyzing the effectiveness of speech enhancement.
[0035] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0036] This method uses frequency domain transformation to efficiently extract feature values of audio information, introduces a one-dimensional convolutional layer to smoothly process the audio information into input terms that are suitable for neural networks, and uses a temporal convolutional network to infer the spatial and temporal relationships of the data, thereby enhancing the useful information and reducing noise information, and significantly improving the intelligibility of audio data.
[0037] This invention introduces a temporal convolutional network to quickly and effectively perform large-scale, complex audio enhancement processing. The structure, number of layers, and parameters of the temporal convolutional network can be learned and adjusted according to actual conditions, making the model flexible and adaptable, thereby further improving the model's performance and effectiveness, and providing stronger support for achieving large-scale, multi-type speech enhancement. Attached Figure Description
[0038] Figure 1 This is a flowchart of the method of the present invention;
[0039] Figure 2 This is a flowchart illustrating the model design process of the method of the present invention.
[0040] Figure 3 Flowcharts are constructed for system metrics.
[0041] Figure 4 This is a schematic diagram of a temporal convolution module;
[0042] Figure 5 This is a diagram of the temporal convolutional network structure. Detailed Implementation
[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the following description will be provided in conjunction with the appendix. Figures 1 to 5The present invention will be described in detail below with specific embodiments. The present invention is a speech enhancement method based on temporal convolutional networks, and the specific steps are as follows:
[0044] Acquire the dataset and preprocess the data to generate data pairs for training the model.
[0045] Obtain the datasets needed to train the model, including clean audio data, noise audio data, room impulse response audio data (RIR), etc.
[0046] Clean data is randomly mixed with noisy data and RIR data to generate an audio dataset with background noise that closely resembles what might happen in reality. The original clean audio data is used as the target value, and the generated data pairs are used for subsequent training.
[0047] Based on general industry knowledge rules and the needs of actual application scenarios, construct the system's measurement standards.
[0048] Based on industry-standard knowledge and rules, commonly used metrics for measuring audio quality are selected as some evaluation indicators for evaluating speech enhancement effects.
[0049] To meet the needs of real-world application scenarios, representative audio quality metrics were selected and combined with the aforementioned general metrics to construct a comprehensive audio quality measurement system.
[0050] (30) Convolutional short-time Fourier transform (ConvSTFT) is used to construct an encoder as the feature value of the audio data and used as the input of the model.
[0051] The short-time Fourier transform of the audio data is calculated to extract the desired audio information as feature values. The calculation formula is as follows:
[0052] (1),
[0053] Where w(t) is a window function, usually a Hanning window or a Gaussian window with the center at zero, and x(t) is the signal to be transformed.
[0054] One-dimensional convolution is introduced to smoothly process audio information into input terms that fit the neural network.
[0055] A temporal convolutional network (TCN) is introduced to train the model, which then decodes the encoder to obtain the speech enhancement result.
[0056] The formula for constructing a causal convolutional layer is as follows:
[0057] (2),
[0058] Where F is the filter, X is a sequence. The above formula indicates that in The causal convolution value at that location.
[0059] Construct extended causal convolutional layers by increasing the number of intervals in the convolutional kernels, i.e., setting the dilation size. The number of intervals increases with the number of layers, and its calculation formula is as follows:
[0060] (3),
[0061] Where F is the filter, X is a sequence. d is the expansion dimension, and the above formula represents that in The causal convolution value with an expansion size of d at the point.
[0062] The weights are normalized, and the formula for calculation is as follows:
[0063] (4),
[0064] Where g is a scalar of control amplitude and v is a unit vector of control direction.
[0065] The activation function ReLU is introduced, and its calculation formula is as follows:
[0066] (5),
[0067] Introduce Dropout.
[0068] The above algorithms are combined into a temporal convolution module, the structure of which is as follows: Figure 4 As shown:
[0069] Multiple temporal convolutional modules are combined sequentially to form a temporal convolutional network, which is used to train the model and obtain audio temporal information. The structure of the temporal convolutional network is as follows: Figure 5 As shown:
[0070] The encoder is decoded to obtain the enhanced audio signal data.
[0071] The effects of speech enhancement were analyzed based on previously established metrics.
[0072] Based on previously established audio quality measurement standards, benchmarks are set, such as commonly used industry models. The performance of commonly used industry models and this model on various evaluation indicators is compared, and the speech enhancement effect is analyzed.
[0073] Compared with traditional methods of speech enhancement using adaptive filters, the advantages of this invention are as follows: This method uses frequency domain transformation to efficiently extract the feature values of audio information, introduces a one-dimensional convolutional layer to smoothly process the audio information into input terms that fit the neural network, and uses a temporal convolutional network to infer the spatial and temporal relationships of the data, thereby enhancing the useful information and reducing noise information, and significantly improving the intelligibility of the audio data.
[0074] Case Study:
[0075] This model uses 360 hours of clean speech data, randomly fusing various noises and reverberations to generate audio data closely resembling real-world scenarios for training. It performs frequency domain processing of the signal data through convolutional short-time Fourier transform to obtain the model input values, and uses a temporal convolutional network for deep data analysis and audio enhancement. Specifically, the number of temporal convolutional layers (num_layer) is set to 3, the number of kernels (kernel_size) is set to 3, the dilation size (dilation_size) is set to 2^i (where i is the number of convolutional layers containing the temporal convolutional module), the stride is set to 1, the padding is set to (kernel_size - 1) * dilation_size, the dropout is set to 0.2, the batch size is set to 256, the epoch is set to 300, the target value is 360 hours of clean audio data, the scale-invariant signal-to-noise ratio (SISNR) is selected as the loss function, the perceptual speech quality assessment (PESQ) is used as the evaluation metric, Adam is used as the optimizer, the learning rate is set to 0.001, and the EarlyStopping is set to 20. After configuring the above parameters, train the model on the training set, obtain feature values and target values by iterating through the training set data, and introduce EarlyStopping to prevent overfitting.
[0076] This model was evaluated using 60 hours of audio data, with a batch size of 256 and an epoch of 300. Scale-invariant signal-to-noise ratio (SISNR) was selected as the loss function, and perceptual speech quality assessment (PESQ) as the evaluation metric. The trained model was used to make predictions on the evaluation dataset to verify its accuracy and stability on datasets different from the training dataset. The performance of the model's speech enhancement was analyzed by combining the loss value calculated from the evaluation loss function and the evaluation value calculated from the evaluation metric.
[0077] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.
Claims
1. A speech enhancement method based on temporal convolutional networks, characterized in that, The method includes the following steps: S1. Obtain the dataset and preprocess the data to generate data pairs for training the model; S2. Based on general industry knowledge rules and the needs of actual application scenarios, construct the system's measurement standards; S3. Construct an encoder using convolutional short-time Fourier transform, which serves as the feature value of the audio data and is used as the model input. S4. Introduce a Temporal Convolutional Network (TCN) training model to decode the encoder and obtain the speech enhancement processing result. S5. Analyze the effect of speech enhancement based on the previously established metrics.
2. The method according to claim 1, characterized in that, S1 specifically includes the following: S11. Obtain the dataset required for training the model; S12. Randomly mix the clean data with the noisy data and RIR data to generate an audio dataset with background noise that is close to what might occur in reality. Use the original clean audio data as the target value and generate data pairs for subsequent training.
3. The method according to claim 1, characterized in that, The specific steps of S2 are as follows: S21. Based on industry-standard knowledge rules, select indicators to measure audio quality as part of the evaluation indicators for evaluating speech enhancement effects. S22. In accordance with the needs of practical application scenarios, scale-invariant signal-to-noise ratio, speech quality perception evaluation index, and short-term objective intelligibility index are selected as audio quality measurement indicators. Combined with the above indicators, a system audio quality measurement index is constructed.
4. The method according to claim 1, characterized in that, The specific steps of S3 are as follows: S31. Calculate the short-time Fourier transform of the audio data and extract the required audio information as feature values. The calculation formula is as follows: （1）， in, The window function is either a Hanning window or a Gaussian window centered at zero. It is the signal to be transformed. For time shift parameters, Angular frequency parameter, This represents the time-frequency representation of the short-time Fourier transform of the audio data. S32. Introduce one-dimensional convolution to smoothly process audio information into input terms that fit the neural network.
5. The method according to claim 1, characterized in that, The specific steps of step S4 are as follows: S41. Construct a causal convolutional layer, the calculation formula of which is as follows: （2）， Where F is the filter, X is a sequence. , The kernel size is the convolution kernel size. for The values of the X sequence at time t are given by the formula above. The causal convolution value at the location; S42. Construct extended causal convolutional layers, increasing the number of intervals in the convolutional kernels. The number of intervals increases with the number of layers, and the calculation formula is as follows: （3）， Where F is the filter, X is a sequence. d is the expansion dimension. The kernel size is the convolution kernel size. for The values of the X sequence at time t are given by the formula above. The causal convolution value at which the dilation size is d; S43. Normalize the weights; the calculation formula is as follows: （4）， Where g is a scalar of control amplitude and v is a unit vector of control direction; S44. Introduce the activation function ReLU, whose calculation formula is as follows: （5）， S45. Dropout is introduced to prevent the network layers from overfitting the audio data used for training; Dropout is a regularization layer that randomly discards the output of neurons with a forward probability p. S46. Combine the above algorithms into a temporal convolution module; S47. Multiple temporal convolutional modules are combined to form a temporal convolutional network, which is used to train the model and obtain audio temporal information. By using multiple convolutional kernels with increasing dilation rates in parallel, multi-scale temporal features are extracted and fused within a single module. S47. Decode the encoder to obtain the enhanced audio signal data.
6. The method according to claim 1, characterized in that, The specific steps of S5 include the following: Selecting three objective evaluation indicators as the evaluation criteria for speech quality, namely, the scale-invariant signal-to-noise ratio (SNR) indicator, the speech quality perception evaluation indicator, and the short-time objective intelligibility indicator. The scale-invariant signal-to-noise ratio indicator evaluates the similarity of the waveform of the enhanced audio data and the clean speech audio data. The larger the value, the higher the similarity, that is, the better the speech enhancement effect. The speech quality perception evaluation indicator evaluates the overall speech quality after the speech enhancement process. The output range of this evaluation standard is between -0.5 and 4.
5. The higher the score, the better the overall speech quality. The short-time objective intelligibility indicator focuses on evaluating the intelligibility of the speech audio. Its value ranges between 0 and 1. The closer the value is to 1, the higher the intelligibility of the speech audio, in order to analyze the effect of speech enhancement.